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FOREWORD: MEDIATIONS OF METHOD 


As the subtitle of the two volumes of Beyond Media Borders: Intermedial 
Relations among Multimodal Media makes clear, these reflections on 
media have the mission to begin where medium-specificity or, what I call 
slightly irreverently, medium-essentialism ends. The media under discus- 
sion here, considered from a great variety of perspectives, are all ‘multi- 
modal’, set in more than one semiotic mode. The most readily 
understandable example we have rehearsed for so long would be, of 
course, cinema or television, the study of which in monodisciplinary 
departments seems to take for granted that they are media, whereas the 
inevitable combination of words and images, colour, sound, narrativity 
and technological effects clearly demonstrates that no single disciplinary 
framework will do. As I am also a maker of films and video, I feel I am in 
a good place to say this. But as the essays in these volumes make clear, 
practically all media deploy more than one modality. 

The point is not so much, however, that ‘multi-’ aspect, although that, 
too, is important, since it advocates an anti-purist view of the media 
products—Lars Ellestróm's term for ‘texts’ and ‘images’, ‘sounds’ and 
‘words’, and what have you, that it is the Humanities’ mission to study. 
What catches my eye is primarily that word ‘relations’, in combination 
with the preposition ‘inter-’, which is particularly dear to me, as I have 
explained more times than I care to remember. Briefly, ‘inter-’ stands for, 
or is, relation, rather than accumulation. It is to be distinguished in crucial 
ways from that currently over-used preposition ‘trans-’, which denotes a 
passage through, without impact from, another domain. With his consis- 
tent interest in media as intermedial and his prolific publication record, 
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many edited volumes, and as director of the Linnaeus University Centre 
for Intermedial and Multimodal Studies (IMS), Ellestróm has become a 
primary authority in that domain that is best characterized as one that 
doesn’t fit any of the traditional disciplinary concepts, yet is probably the 
largest, most frequently practised mode of communication among humans, 
indispensable for human life. Ellestróm's ongoing focus on—his intellec- 
tual loyalty to—the idea of the semiotic, a concept and field that on its own 
already indicates the need for the ‘beyond’ in the books’ main title, dem- 
onstrates a resistance to ephemeral academic fashion and a consistency of 
thought without dogmatism which I consider characteristic of the semi- 
otic perspective. Briefly again, a semiotic perspective asks how we make 
meaning. The interest of these volumes lies in the importance of commu- 
nication in general, without which no human society is possible. 

Media, as the editor explains, are always-already ‘inter-’, as the century- 
old debates about inter-arts clearly demonstrates. The preposition is a 
bridge, and the articles brought together here explore what the bridge 
bridges. This requires reflection on the concept of media itself. One can- 
not understand intermediality without a sense of what a medium is; even 
if, as such, in its purity, it doesn’t exist. With exemplary clarity, Ellestróm 
begins his substantial opening and synthesizing article with five tendencies 
he finds damaging for intellectual achievement in (inter)media studies. 
Anyone interested in this field of study will recognize these tendencies and 
agree with the editor's critique of them. But then, the challenge is how to 
remedy these problems. This is where Ellestróm earns his authority: he 
proceeds to announce how these tendencies will be countered, or over- 
come, in the present volumes. If only all academics would take the time 
and bother to lay out what they are up against and then redress it: aca- 
demic bliss would ensue. In other words, this is real progress in the collec- 
tive thinking of cultural analysis. Felicitously refraining from short 
definitions, he embeds the relevant concepts in what he calls a ‘model’, 
but what those of us with a mild case of *model-phobia'—the fear of a 
certain scientistic demand of rigour before all else—may also see as a theo- 
retical frame. Felicitously, he calls his activity ‘circumscribing’ rather than 
defining. His approach alone, then, already demonstrates in the first pages 
of his long introductory text an academic position that integrates instead 
of separating creativity and rigour and thus not only helps us understand 
the general principles of communication but, through detailed analysis, 
makes us ‘communicationally intelligent’, if I may follow discursively the 
example of psychoanalyst Christopher Bollas who, in his 1992 book Being 
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a Character, sensitizes us to the complexities and thereby, clarity, of how 
people are able, and by the media products, enabled, to communicate 
effectively, with nuance. 

There is not a term or concept here that is not both circumscribed and 
relativized and put to convincing use. The length of the introductory essay 
is, in this sense, simply a demonstration of generosity. For example, the 
central concept of ‘transfer’ that we can hardly do without when talking 
about communication is neither defined in a simplistic way, as a postal 
service that goes in one direction only, nor theorized into incomprehensi- 
bility. The idea of transferring means that a message goes from a sender to 
a receiver; we were told in the early days of semiotic theory. Of course, in 
order to discuss communication, we must consider the idea that a message 
is indeed transferred from a sender to a receiver; without it, we are floun- 
dering. In this, Ellestróm is realistic; he doesn’t reinvent the wheel. Yet, 
the implicit (but not explicit) notion that the content of a message, as well 
as its form, go wholesale from sender to receiver, as endorsed in traditional 
semiotic theory, is clearly untenable. For, the sender’s message, with the 
sender always already ‘in’ communication, will always be influenced, or 
coloured, by what the sender expects, and has reasons to expect, the 
receiver will wish, grasp, appreciate. 

What do we do, then? Instead of casually rejecting the idea, concept or 
notion, Ellestróm and his colleagues in these volumes recalibrate and 
nuance what we consider a message to be, with the help of the relationality 
that the preposition ‘inter-’ implies. This makes the sender-message- 
receiver process an interaction, mutually responsive, hence, communica- 
tive in the true sense. The change from ‘sender’ to ‘producer’ intimates 
that the former sender has made something. The former ‘receiver’ has 
shed her passivity by becoming a ‘perceiver’, a term that adds the activity 
performed at the other end of the process. And when the term ‘meaning’ 
is hurt by a long history of rigid semantics, as is the case of many of the 
concepts we use as if they were just ordinary words, they come up with 
alternatives, but not without bringing these in ‘discussion’ with the sim- 
pler but problematic predecessors. The need for a concept that cannot be 
reduced to dictionary definitions compels the authors, guided by the 
experienced and ingenious editor, to come up with richer terms that are 
able to encompass all those nuances that were always a bit bothersome and 
that we liked to discard or ignore. Thus, ‘cognitive import’ cannot be 
reduced to ‘meaning’, and neither can it be confined to language. That 
would make the substitution of a well-known term by a new one pointless. 
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Instead, the new term necessarily includes the embodied aspect of com- 
munication. This eliminates the mind-body dichotomy to which we are so 
tenaciously attached; not because we believe in it, but because, until these 
volumes, we had no alternative vision. 

The word ‘dichotomy’, here, is perhaps the most central opponent in 
these volumes’ discourse. And as with ‘inter-’ as implying relationality, I 
feel very close and committed to an approach that does not take binary 
opposition as its ‘normal’, standard mode of thinking. And once we are 
willing to give up on dichotomies such as mind/body, it becomes possible 
to complicate all those dichotomies that structure what we have taken for 
granted and should let go in order to recognize the richness of mental 
life—mental in a way that does not discard the body but endorses it, along 
with materiality, as integrally participating in the thinking that communi- 
cation stimulates, helps along and substantiates. Both the partners in com- 
munication, who can be singular and, at the same time, plural, and the site 
of communication, are necessarily material or bound to materiality. 
Moreover, the sense-based nature of communication makes the abstract 
ideas surrounding communication theory, not only untenable, but futile, 
meaningless. Getting rid of, or at the very least, bracketing, binary opposi- 
tion as a way of thinking is for me the primary merit of the approach pre- 
sented here. 

So, the first thing these books achieve is to complicate things, in order 
to get rid of cliché simplicity, and then, right after that, to clarify those 
complicated ideas, concepts and the models that encompass them. This is 
perhaps the most important merit of these volumes. They complicate what 
we thought we knew and clarify what we thought is difficult. With that 
move as their starting point, the enormous variety of topics of the chapters 
become, thanks to the many cross-references from one article to another, 
a polyphony. Rather than a cacophony of loud divergent voices, this 
polyphony constitutes a symphony that, as a whole, maps the enormously 
large field of the indispensable communication that is human culture, 
without pedantically demanding that every reader be an expert in all those 
fields. I don’t think anyone can master all the areas presented and exam- 
ined in the contributions, but the taste of it we get makes us at the very 
least genuinely interested. This is not a dictionary or an encyclopaedia but 
a beautifully crafted patchwork of thoughts. 

The conceptual travels are stimulating, never off-putting, because they 
are completely without the plodding idiosyncrasies one so often encoun- 
ters when new concepts are proposed. And devoid of the polemical 
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discussions with other terminologies, well explained and labelled mean- 
ingfully, the conceptual network towards which these books move fills 
itself as we read along and thus ends up offering a ground for cultural 
analysis that I am eager to put my feet on. Solid, reliable and, still, excit- 
ing. What more would we wish from in-depth academic work? This collec- 
tive, collaborative work is based on a deep understanding of what scholarly 
work should be: an act of communication between producers and perceiv- 
ers, as the view presented here would have it, one that makes its readers 
feel involved. This is the only way they can learn something new. 


University of Amsterdam Mieke Bal 
Amsterdam, The Netherlands 


PREFACE 


In 2010, Palgrave Macmillan published a volume entitled Media Borders, 
Multimodality and Intermediality, which I had the pleasure to edit. It 
included my own rather extensive introductory article, “The Modalities of 
Media: A Model for Understanding Intermedial Relations’, which has 
since then attracted some attention in intermediality studies. It is my most 
quoted publication, and scholars and students still use the book and my 
introductory article in research and education. For my own part, I apply 
the core concepts of ‘The Modalities of Media’ as a basis for all my 
research, including in the two Palgrave Pivot books Media Transformation 
(2014) and Transmedial Narration (2019). Over the last decade, how- 
ever, I have also deepened, developed and slightly modified the original 
ideas because I think some of them were formulated prematurely. I have 
also noted that people sometimes misunderstand certain parts of the arti- 
cle because of my somewhat inadequate and occasionally confusing ways 
of explaining some of the concepts. Therefore, I decided to rewrite "The 
Modalities of Media: A Model for Understanding Intermedial Relations’. 

However, the reworking became more substantial than I had expected, 
resulting in a text that is not only modified and updated but also signifi- 
cantly expanded, incorporating ideas that I have presented in other publi- 
cations during the last decade. Therefore, I have called it "The Modalities 
of Media II: An Expanded Model for Understanding Intermedial 
Relations’. The new version more clearly frames mediality and intermedi- 
ality in the context of inter-human communication and defines the central 
concept of media product as the intermediate entity that makes communi- 
cation among human minds possible. It retains, but slightly modifies and 
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expounds, the central idea of characterizing media products in terms of 
four media modalities, four kinds of media traits. For instance, the discus- 
sions now include not only virtual (represented) time and space but also 
virtual (represented) materialities and sensory perceptions. Providing a 
fuller picture of representing and represented media traits, as well as add- 
ing discussions of cross-modal cognitive capacities of the human mind, 
makes it possible to offer a much-developed understanding of the con- 
cepts of media types and media borders and what it means to cross media 
borders. As a result, the new article hopefully better explains the intricacies 
of media integration and media transformation. Overall, most of the con- 
cepts have been fine-tuned, leading to a more consistent and developed 
framework. However, attentive readers will note that I have not men- 
tioned a few ideas that I briefly discussed in the original article. This does 
not necessarily mean that I have abandoned them; rather, I have decided 
to develop them further in other publications instead of trying to squeeze 
even more into an already extensive article. Nevertheless, “The Modalities 
of Media IP? is supposed to replace rather than complement the original 
article. 

This means that the two-volume Beyond Media Borders: Intermedial 
Relations among Multimodal Media is effectively a completely new publi- 
cation. All of its other contributions are entirely novel compared to Media 
Borders, Multimodality and Intermediality (2010) and are written by 
authors that (with only one exception) are not the same as those in the 
earlier book. The main idea of the new publication is not only to launch 
an updated version of "The Modalities of Media’ but also to present it 
together with a collection of fresh articles written by scholars from a broad 
variety of subject areas, united by their references to the concepts originat- 
ing from “The Modalities of Media’. 

Besides being highly original pieces of scholarship in themselves, the 
accompanying articles practically illustrate, exemplify and clarify how the 
concepts developed in “The Modalities of Media IP can be used for 
methodical investigation, explanation and interpretation of media traits 
and media interrelations in a broad selection of old and new media types. 
To provide space for analysis of such a wide range of dissimilar media 
types, without reducing the complexity of the arguments, two volumes are 
required. Their title, Beyond Media Borders: Intermedial Relations among 
Multimodal Media, reflects the underlying idea that all media types are 
more or less multimodal and that comparing media types requires that 
these multimodal traits being analysed and compared in various ways. As 
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different basic media types have diverging but also partly overlapping 
modes (for instance, several dissimilar media types have visuality or tempo- 
rality in common), and because humans have cognitive capacities to partly 
overbridge modal differences (between, for instance, space and time or 
vision and hearing), media borders are not definite; in that sense, one 
must move ‘beyond’ media borders. 

Overall, the two volumes form a collection with strong internal coher- 
ence and abundant cross-references among its contributions (not only to 
‘The Modalities of Media IT). Simultaneously, they cover and intercon- 
nect a comprehensive range of very different media types that scholars 
have traditionally investigated through more limited, media-specific con- 
cepts. Hence, the two volumes should preferably be read together as a 
unified, polyphonic and interdisciplinary contribution to the study of 
media interrelations. 


Växjö, Sweden Lars Ellestróm 
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ABOUT THE BOOK 


Although all of the contributions can be read as separate articles, the two 
volumes of Beyond Media Borders form a whole. Because the contributions 
are written in concert and include some dialogues, reading the publication 
in its entirety adds substantial value. Part I in Volume 1, “The Model’, 
contains the extensive theoretical framework presented in “The Modalities 
of Media II: An Expanded Model for Understanding Intermedial rela- 
tions’. Part II in Volume 2, “The Model Applied’, offers a brief summary 
and some elaborations that end the two volumes. Between these two 
opening and closing parts, one finds Part II in Volume 1, ‘Media 
Integration’, and Part I in Volume 2, ‘Media Transformation’, which con- 
tain the majority of contributions. As explained in ‘The Modalities of 
Media IJ’, media integration and media transformation are not absolute 
properties of media and their interrelations, but rather analytical perspec- 
tives. Hence, the division of articles into two parts only reflects dominant 
analytical viewpoints in the various contributions; a closer look at them 
reveals that they all, to some extent, apply an integrational as well as a 
transformational perspective. 

This is the second volume of Beyond Media Borders. The complete table 
of contents for both volumes is as follows: 
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1.1 INTRODUCTION 


In recent decades, a flurry of excellent scholarship has described media 
and processes of mediation in increasingly expansive terms. Media, much 
of this work posits, is not only the specific technologies by which informa- 
tion is disseminated, but also any communicative conduit that conveys 
ideas or meaning between one place or person and another. As Lars 
Ellestróm described in his original “Modalities of media” article and fur- 
ther elucidates in his essay in this volume, for example, media products— 
phenomenon, objects, bodies, and/or extensions of bodies—transfer 
“cognitive imports” between the “producer’s mind,” where they 
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originated, consciously or unconsciously, and the “perceiver’s mind,” 
where they are received bearing some resemblance to their original state 
(Ellestróm 2020: 12-13). Ellestróm's and other scholarship then elabo- 
rate and concretize these models with examples: a dancer develops a con- 
cept, generates choreography, and uses their body as a medium of display 
upon and through which audience members see, hear, and feel some ver- 
sion of that original concept. A photographer sees a scene, decides how to 
frame and compose it, and captures it on film; the digital photo or print 
enables viewers to glimpse a version of that original scene. For obvious 
reasons, these examples almost always involve successful communication: 
the viewer gets a sense of what is pictured, the photographer deems their 
captured image acceptable, the audience is able to make sense of the 
dance, and the choreographer and dancer both feel as if they have shared 
something with those present. 

However, media products—and particularly intermedial products—are 
not always able to transmit meanings and ideas so smoothly. And at times, 
media products are envisioned and created precisely so that they will likely 
not transmit meanings and ideas in a straightforward way. In this essay, I 
explore a series of historical and contemporary media objects and perfor- 
mances that do not necessarily facilitate a ^successful" transfer of meaning, 
in part because of their intermedial configurations: a silent film about (and 
featuring repeated performances by) a ventriloquist, another that revolves 
around about an opera singer's stage debut, radio broadcasts of film pro- 
grams in the 1920s, and finally, a modern American theater production 
that relays a narrative through a series of media practices rooted in histori- 
cal moments, geographies, and languages distant from most audience 
members’ experiences. Each of these media objects and performances mix, 
combine, and transpose multiple media. In doing so, each conceals as well 
as reveals: something is lost or obscured in translation.! At times, these 
“losses” are purely accidental, the result of an overly ambitious agenda: 
aesthetics and/or content associated with one medium are transposed to 
another that cannot quite contain them. At other times, these losses are 
the natural byproduct of experimentation and a still-developing under- 
standing of a medium's modalities, possibilities, and limits: a creator rep- 
resents a media object in a technical medium of display defined by strikingly 
different modalities to see how both media will be reimagined, and their 
relationship to one another reconfigured. At still other times, these losses 
are entirely purposeful: an aesthetic, social, or cultural intervention 
designed to yield specific effects. Yet in all cases, these acts of concealment 
are—or become—meaningful and productive. 
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That a media product is likely better suited to one technical medium of 
display than another is hardly a novel concept. Nor is it news that some 
media products are more easily representable in certain media, nor that 
media require contextual frameworks and experiences. Indeed, as Ellestróm 
notes in his original *Modalities of media" essay, while technical media are 
generally able to mediate a range of different media, a television show will 
more “completely” and “successfully” mediate a film than a painting 
could, because its modal capacities are far more like those of film (Ellestróm 
2010: 31). In this essay, I am interested in expanding our understanding 
of what constitutes “successful” mediation, particularly in the context of 
intermediality. Numerous scholars have argued that intermediality is 
defined in part by its resistance to completeness and unity. Irina Rajewsky, 
for example, describes intermedial references as “as-if” situations: though 
one medium is treated as if it is another, it never actually becomes another. 
While that medium may “generate an illusion of another medium's spe- 
cific practice,” there remains an “intermedial gap” between the attempted 
enactment or reproduction of the medium and the medium itself (Rajewsky 
2005: 55). These gaps and “as ifs” are often made readily visible; indeed, 
as Peter Boenisch writes, the broadcasting of these *detours, inconsisten- 
cies, and contradictions" refuses our easy immersion into a narrative or 
world and instead draws our attention to the (inter)media at work 
(Boenisch 2006: 115)? By creating space for and highlighting these 
gaps—these moments that withhold as much as they communicate, and 
that communicate withholding—artists and performers generate a wealth 
of new expressive possibilities. These new modes of expression, in turn, 
promise new affective and perceptual experiences for performers and audi- 
ences alike. And at times, the creation of moments in which media do not 
or are not able to communicate smoothly also prompts new aesthetic and 
technical innovations, reshaping genres and media, and drawing attention 
in our cultural imagination to media's plasticity and possibility. 


1.2 . VENTRILOQUISM WITH No VOICE 


The Unholy Three, a 1925 silent film directed by Tod Browning, is a sus- 
penseful thriller about an elaborate criminal ruse hatched by a former side- 
show ventriloquist, Prof. Echo (Lon Chaney), and two of his colleagues, a 
little person named Tweedledee (Harry Earles) and a strongman named 
Hercules (Victor McLaglen).? Using his vocal mimicry skills and a clever 
costume, Echo disguises himself as Mrs. O'Grady, the proprietress ofa pet 
shop specializing in talking parrots, and the elderly grandmother of his 
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pickpocket girlfriend, Rosie O’Grady (Mae Busch). “Mrs. O’Grady” uses 
ventriloquism to convince wealthy customers that the birds in her shop 
can speak on command. The sales are part of a larger ruse, though. When 
customers call to complain that their new pets will not talk, Mrs. O’Grady 
makes house calls; while she convinces the customers that the birds can 
actually speak, her accomplice Tweedledee, disguised as a napping baby in 
a pram, is left alone to case the house so the trio can later return and rob it. 

The conundrum bound up in depicting this narrative on the silent 
screen is immediately obvious. Ventriloquism is predicated on aural illu- 
sion. The ventriloquist speaks with no movement or other visible sign, 
throwing their voice in order to vocally animate that which does not speak. 
Without audible voices, there is no magic: the ventriloquist need not speak 
with still lips at all, much less throw their voice or disguise their vocal tim- 
bre and speech patterns. How to depict ventriloquism and vocal imper- 
sonation in the absence of precisely the modalities that define them? Or, 
to borrow from Kate Newell’s essay in this volume, how to “reframe” 
ventriloquism to be seen and heard as it moves from stage to the silent 
screen? (Newell 2020). Previews for the film puzzled over the question. As 
a writer for Picture-Play Magazine wrote, “when the parrots in his shop 
can't talk Chaney is supposed to imitate them with his ventriloquial tal- 
ents, and they are probably having a monkey-and-parrot time of it trying 
to register this on the screen” (Hollywood Highlights 1925: 114). 

The film’s creators employed a variety of narrative, dialogic, and visual 
cues to represent ventriloquism and vocal disguise. From its opening 
moments, the film establishes Echo’s identity as a ventriloquist: we meet 
him performing with a dummy at a sideshow. Though he quickly begins 
ventriloquizing birds instead, the dummy remains a conspicuous prop 
throughout the film, constantly reminding us of his skill. Additionally, the 
film deploys physical markers and intertitle dialects to visually convey 
Echo’s ability to successfully alter his voice. When disguised as Mrs. 
O’Grady, Echo’s bent posture, relatively still lips, gently tilted head, and 
relaxed cheeks and chin suggest a quiet, gentle voice; her dialogue that 
appears in intertitles is correspondingly mannered and mild. When Echo 
breaks his Mrs. O’Grady character or appears out of disguise, however, his 
face becomes far more animated, his eyebrows raising and lowering for 
emphasis as his speech emanates from the corner of his slightly distorted 
mouth. These physical hints are registered in the intertitle dialogue with 
coarser expressions, slang, and contractions. Frustrated at Rosie’s flirting 
with the naive and besotted pet store clerk, Hector (Matt Moore), for 
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example, Echo rips off his Mrs. O’Grady wig, juts his head forward, and 
wags his finger in her face, disgustedly exclaiming via an intertitle, “You’re 
makin’ a play for that guy!” The film also draws our attention to the sound 
of voices through offhanded diegetic comments. In one of the early pet 
store scenes, for example, Mrs. O’Grady tries to shoo Rosie away from 
Hector by remarking on the effect of Rosie’s voice on “baby Willie” 
(Tweedledee in his disguise), who fusses on the floor nearby. “Little Willie 
darling heard your voice,” the older woman tells the younger. “He wants 
you to take him bye-bye.” Neither these explicit references to character’s 
voices, nor the careful visualization of speech through intertitles and facial 
expression are accidental: both tacitly encourage audiences to imagine the 
sounds of these different voices. 

As the reviewer quoted above predicted, some of the most striking 
attempts to visualize Echo’s ventriloquism utilize the parrots. During the 
very first scene in the pet store, Hector comments on Mrs. O’Grady’s 
special ability to make the birds talk. When a customer enters and begins 
looking at the birds, Hector immediately runs to summon Mrs. O’Grady, 
telling her (via intertitle), “I think you’d better show the parrots. They 
talk so well for you.” As Mrs. O’Grady obliges, the action is interrupted 
by an intertitle reminding the audience of Echo’s ventriloquial proficien- 
cies and likening the parrots to Echo’s dummy: “Echo, the ventriloquist, 
could make wooden dummies talk—now watch the jolly parrots!” The 
film immediately cuts to a shot of Mrs. O’Grady, surrounded by caged and 
loose parrots; as each bird “speaks,” speech bubbles appear over their 
heads containing intertitles depicting the bird’s words: “Good morning, 
Aunty!” “Hello Polly!” “Pretty Lady!” (see Fig. 1.1). The shot cuts to 
Hector, who again draws attention to Mrs. O’Grady’s magical ability to 
make the birds talk when he raves to Rosie, “Isn’t it wonderful how the 
parrots always talk to your Grandma?” 

Echo’s ventriloquism is a crucial plot device—the mechanism by which 
the trio’s crimes are facilitated. However, it is also central to the film’s 
climactic scene, and its narrative resolution. Increasingly uncomfortable 
with the trio’s activities, Rosie falls in love with Hector. When the trio 
botches a robbery, Echo frames Hector and then kidnaps Rosie in hopes 
of reigniting their former passion. But he cannot bear to see Rosie so mis- 
erable, and sneaks into Hector’s trial in hopes of exonerating him through 
a final ventriloquism performance. Echo manages to slip Hector a note 
instructing him to take the stand and silently repeat the Lord’s Prayer over 
and over, moving his lips without actually making a sound. As Hector 


8 M.SIMONSON 


Fig. 1.1 Mrs. O’Grady and her “talking” parrots in The Unholy Three (dir. Tod 
Browning, 1925). All rights reserved 


follows the note’s instructions, the film’s audience is treated to yet another 
visual strategy for representing ventriloquism on the silent screen: the 
camera oscillates between close-ups of Echo in the gallery, his throat and 
jaw moving slightly as he envoices Hector; first person intertitles exonerat- 
ing Hector; and shots of Hector obediently moving his lips. Seconds later, 
Hector draws attention to Echo’s ventriloquism when he declares, “That 
wasn’t me talking. I didn’t say a word.” In this moment, ventriloquism is 
portrayed by revealing—and commenting on—its mechanics. 

While the scene punctures the ventriloquial illusion for the film’s audi- 
ence, the performance goes off without a hitch within the film’s diegesis. 
Neither the judge nor either of the lawyers acknowledge Hector’s claim 
that he has just been envoiced—an act that presumably amounts to per- 
jury. Nor do any of them seem to doubt the contents of his ventriloquized 
testimony. However, even this diegetically “successful” ventriloquial per- 
formance is not enough to clear Hector’s name. The judge immediately 
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instructs the jury to disregard Hector’s testimony on the grounds that 
there are no witnesses to confirm this new story, and sends them out to 
deliberate. Narrative closure comes only when Echo intervenes and makes 
a full confession. In the end, ventriloquism can only get so far on the 
silent screen. 

As was common practice, a talkie remake of The Unholy Three (directed 
by Jack Conway) was released by MGM in 1930, shortly after the wide- 
spread conversion to sound film. The remake retained the silent film’s 
narrative conceit and characters, and both Lon Chaney and Harry Earles 
reprised their roles alongside an otherwise new cast. It also retained ven- 
triloquism as a drawing card, central plot device, and Echo’s defining tal- 
ent. Advertisements and previews for the new Unholy Three touted the 
imminent logic of making a sound version of this particular film. A writer 
for Film Daily noted that the story was “a bigger draw in sound, for so 
many of the dramatic incidents depend on audible effects, such as the 
ventriloquist’s dummy,” and advised exhibitors to highlight Chaney’s 
“varied speaking voices” in their pitches to audiences (The Unholy Three 
1930b: 10). With sound, not only could audiences marvel at Chaney’s 
vocal impersonations, they could also hear his voice magically emerge 
from his dummy’s flapping mouth as his own lips remained remark- 
ably still. 

However, though sound film includes an aural channel, it is in some 
ways no more appropriate for the display of ventriloquial acts than silent 
film: it does not replicate the modalities that define ventriloquism and 
vocal impersonation, either. Ironically, it is because sound film is itself a 
ventriloquial medium that it cannot convincingly represent ventriloquism.* 
Watching sound films, audiences are invited—indeed, compelled—to pre- 
tend that the sound they hear issues directly from the image that they see 
on screen. In reality, though, these synchronized images and sounds are 
usually the result of careful construction, re-recording, dubbing, and 
extensive editing. As film sound scholar Michel Chion notes, the material- 
ity of film requires “an incision or cut between body and voice” that is 
then covered over enabling us “to believe, implicitly or explicitly, that the 
body and voice cohere in some self-evident, natural way” (Chion 1999 
[1982]: 125-126). By 1930, MGM was certainly using such techniques; 
and thanks in part to decades of cinematic “attractions” that openly dis- 
played all sorts of editing tricks, many audience members attending The 
Unholy Three were savvy enough to understand that Lon Chaney’s 
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ventriloquial stunts and vocal disguises could easily have been generated 
through technical means. 

Readily anticipating audience skepticism, MGM staged a publicity stunt 
to reassure audiences that it was actually Chaney’s voice they were hear- 
ing, and not a cast of vocal extras dubbed in: Chaney issued a signed affi- 
davit that read, in part, “The ventriloquist’s, the old woman’s, the 
dummy’s, the parrot’s, and the girl's are actual reproductions of my own 
voice and in no place in said photoplay or in any of the various characters 
portrayed by me in said photoplay was a ‘double’ or substitute used for my 
voice.” (Inside stuff—Pictures 1930: 49; The Unholy Three advertise- 
ment 1930: 26). The affidavit was widely reported in film trade press, fan 
magazines, and daily newspapers, and was reproduced in advertisements 
for the film. Notably, however, the affidavit—and the press surrounding 
it—blatantly sidesteps larger questions about the film’s representations of 
ventriloquism. The affidavit pointedly does not claim that Chaney’s vocal 
impersonations were synchronously recorded with the film’s image track, 
nor that he ventriloquized his dummy or the parrots live on set. While his 
vocal impersonation skills were validated, then, little was done to convince 
audiences of the validity of his ventriloquial performances. 

Strikingly, the most convincing display of vocal trickery in the talkie 
version of The Unholy Three is achieved through failure. One of the few 
moments in the talkie that significantly departs from the silent version 
comes in the final courtroom scene. Instead of ventriloquizing Hector 
from the audience, Echo actually takes the witness stand in his Mrs. 
O’Grady disguise. As he testifies, his vocal disguise slips: Mrs. O’Grady’s 
voice descends a bit, then a bit more, and eventually is so low that the 
defense attorney approaches her and rips off her wig, revealing the whole 
ruse. Unmasked, Echo confesses and is found guilty and sentenced. While 
the silent version ends with the failure of ventriloquism, the sound version 
culminates in actual ventriloquial failure. The authenticity of Echo’s vocal 
trickery is confirmed by showcasing its dissolve. 

The silent and sound versions of The Unholy Three make a counterintui- 
tive proposition: to represent ventriloquism using a medium—film—that 
at best obscures and at worst undermines the illusion that defines the art. 
While these films develop strategies to do so, the fact remains that silent 
and sound films both lack the modal configuration to “completely” medi- 
ate a ventriloquial performance—or, to put it differently, ventriloquism 
exceeds the modal capacities of the film projector and screen. Despite this 
modal mismatch, though, the two Unholy Three films are hardly failures. 


1 FINDING MEANING IN INTERMEDIAL GAPS ll 


Lon Chaney was famed for the range of characters he convincingly embod- 
ied: reviewers and fans marveled at his transformation from one film to the 
next. Playing a ventriloquist offered Chaney the opportunity to embody 
radically different characters within the same film, and at times, within the 
same scene. Reviews and commentary on the silent film highlight Chaney's 
marked visual transformations as he shifted from Echo to Mrs. O'Grady, 
as well as his expressive abilities within each of the roles (The Shadow 
Stage 1925; The Unholy Three 1925a: 45; The Unholy Three 1925b: 
845). Describing the film's final moments, a writer for Exbibitor's Trade 
Review noted that though Echo lightly bid Rosie goodbye *with a laugh 
that is akin to mockery" after his confession, within seconds of her depar- 
ture, “his seamed face is limned with the searing lines of heartbreak” (The 
Unholy Three 1925a: 45). Similarly, the sound version enabled audiences 
to hear Chaney’s voice for the first time and treated them to Chaney’s 
vocal impersonation abilities. Long nicknamed “the man of one thousand 
faces,” the sound version of The Unholy Three led the press to embrace a 
new moniker: *the man of one thousand voices" (Nelson 1930: 32; Moak 
1930: 44). While Chaney's transformations had in the past hinged on 
costumes, makeup, and his acting abilities, Te Unholy Three films first 
implied and then revealed that his disguises and transformations were 
audible, as well: he could fake convincing laughter, shift his vocal timbre, 
and alter his speech patterns. Here, Chaney dazzled with his ability to shift 
in a moment from one emotion to the next, one voice to another, one 
person to a different person. 

Shoehorned into the film's narrative, ventriloquism also prompted 
innovation within the cinematic medium and the film industry more 
broadly. The creators of the silent Unholy Three explored how the cine- 
matic medium might visualize vocal production and illusion, as well as the 
acts of hearing and mishearing; in doing so, they reconfigured and reimag- 
ined silent cinema's sensorial modalities. Some of the strategies used to 
represent ventriloquism—intertitles, moving lips and expressive faces, 
close-ups and short/reverse shot sequences—align with silent film's usual 
techniques for conveying speech. Other strategies might themselves be 
considered ventriloquial: speech bubble text and oscillating throats speak 
*for" and replace these time-tested techniques.? In the sound remake of 
The Unholy Three, too, Chaney's ventriloquist character prompted the 
film’s creators, MGM, and even the press to negotiate and consider how 
best to acknowledge and address audience awareness of sound cinema’s 
mechanical and technical underpinnings—precisely in order to enable 
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audiences to suspend their disbelief or even disavow film’s status as a 
highly constructed medium. The intermediality of both the silent and 
sound versions of The Unholy Three, then, generates successful medial 
experimentation, recombinations, and fluidity, beginning to reimagine, 
challenge, and construct the defining modalities and medial limits of silent 
and sound film. 


1.3. STENT FILM WITH No IMAGE 


In the first years of the 1920s, radio, which had been limited to military 
use during World War I, emerged as a highly popular and much debated 
technology. Amateur radio operators and new commercial radio experi- 
ments flooded the airwaves, broadcasting a range of different program 
formats, content, and musical styles. Among those who embraced and 
helped to shape the emergent medium were film studios and exhibitors: 
radio was viewed as an ideal venue for advertising new pictures, a sort of 
live fan magazine by which movie stars could directly address their adoring 
audiences, and a potential source for film accompaniment—if not a means 
of making films “talk.” 

While many in the industry eagerly explored radio's possible applica- 
tions, Samuel “Roxy” Rothafel, then the manager of the Capitol Theater 
in New York City, was particularly visionary. In 1922, he began broadcast- 
ing concerts by the Capitol Theater orchestra over WEAF, and hosting 
shows that promoted the theater's film programs and featured perfor- 
mances by a rotating cast of orchestra members and the other performers 
who appeared in theater presentations before and between films. By April 
of 1923, Roxy's radio broadcasts were synchronized with his evening the- 
ater program—or rather, as media scholar Ross Melnick has argued, the 
Capitol's evening theater program was dictated by the broadcast schedule 
(Melnick 2012: 212).° Period articles in Radio Digest and other trade 
publications reported that at 7:20 pm, Roxy would introduce the eve- 
ning's broadcast from near the Capitol Theater's stage (but out of ear- and 
eye-shot of the audience in the theater), sharing details about the orches- 
tra's musical selections and other things they would hear in the two hours 
ahead (Lanzius 1923: 5). Then, at 7:30 pm, the radio audience would be 
invited to listen along with the theater audience to the orchestral overture 
that traditionally opened silent film presentations in movie palaces. But 
radio listeners were not just allowed to listen in on the theater's musical 
performances. As the newsreel and other short films appeared on the 
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Capitol’s screen following the overture, the radio audience heard Roxy’s 
careful descriptions of each film’s narrative and visual imagery over the 
accompanying music. Then, as live performers appeared on the theater’s 
stage between the short subjects and the evening’s feature, Roxy intro- 
duced each and, when necessary, described precisely what the theater’s 
audience was seeing, offering audiences mental images to compliment the 
musical performances and other acts that they heard through the ether. 
Shortly after 8:00 pm, as the feature film began, Roxy and his “Gang” of 
performers moved to the theater’s broadcasting studio and continued 
their broadcast with further conversation and additional musical perfor- 
mances until 9:00 pm. 

At first glance, these radio broadcasts, and particularly the segments 
broadcasted live from the theater’s stage, seem as counterintuitive as a 
silent film about ventriloquism. Here too, media objects—silent films and 
live stage performances—were being mediated by a technical medium of 
display—radio—with significantly different modal characteristics. It is 
tempting to read these broadcasts as a case of remediation gone wrong: 
the nascent radio industry's attempt to take up and incorporate the “older” 
medium of silent film in order to draw legitimacy from and ultimately 
displace it.” And certainly, the broadcasts followed the remedial logic of 
claiming (and benefiting from) the status of silent film and the artists who 
work in it, much as silent film did with opera, as is discussed in the next 
section (Bolter and Grusin 1999: 69). Broadcasts of silent film presenta- 
tions borrowed and represented silent film's familiar entertainment for- 
mats, compelling ready-made narratives, and well-prepared musical and 
stage performances. They also attempted to invoke, if not create, the 
atmosphere of the silent film theater in listeners’ living rooms. As Radio 
Digest writer Evelyn Lanzius reported in 1923, “One evening recently, 
previous to the rendering of ‘Love’s Old Sweet Song’ by ‘Betsy,’ Mr. 
Rothafel requested his Radio audience to lower their lights, thus bringing 
his own lighting effects into their homes. Many wrote in to say that they 
had followed this suggestion with telling effect? (Lanzius 1923: 5). 

Even as these radio broadcasts remediated silent film, they also distin- 
guished themselves from it. On air, Roxy sought to create an intimacy that 
was impossible at film screenings in his enormous Capitol Theater. His 
patter was familiar, his descriptions accessible and casual. He presented 
himself as a personal friend of his listeners, and encouraged the musicians 
and performers who joined him in the studio to do the same. He regularly 
introduced and referred to his musical guests by first names and 
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nicknames: conductor William Axt was called Billy on air, singer Betsy 
Ayres was simply Betsy, and violinist and concertmaster Eugene Ormandy 
was often presented as *The Blue Blonde." The featured performers chat- 
ted casually with the listening public, generating both public personae and 
imagined relationships that were nearly impossible in a darkened film the- 
ater. As Lanzius wrote in Radio Digest, “Roxie [sic] introduces each artist 
most informally, and encourages them to talk in little personalities to the 
invisible audience, these being mostly in humorous vein, and there is no 
doubt that there is an unusual sympathy and bond of interest between the 
artists of the Capitol broadcasting studio and the Radio listeners? (Lanzius 
1923: 5). While radio lacked the visual dimension so central to silent film 
and the live performances presented alongside, its modalities and the 
social and cultural practices that surrounded it enabled radio to exceed 
silent film in meaningful ways. 

It is important to remember, however, that Rothafel’s broadcasts were 
driven as much by Rothafel himself—a fixture within silent film exhibi- 
tion—as they were by those in the radio industry. Even if those at WEAF 
and its parent company hoped that the venture might legitimize and allow 
radio to compete within the entertainment landscape, and some within 
the film industry engaged in a fair amount of hand-wringing over radio’s 
potential to decrease filmgoing, Rothafel and other exhibitors who broad- 
cast from their theaters clearly thought of the practice as a means of renew- 
ing and expanding silent film’s cultural relevance. Period audiences, too, 
seem to have understood that listening to silent film via radio was pro- 
foundly different than attending a film presentation at the theater; that is, 
the technical medium of display profoundly changed the media product 
(Ellestróm 2020: 33-40). Indeed, audiences in the 1920s likely used the 
broadcasts as a means to imagine the original media objects and 
performances.? 

Rothafel's broadcasts hardly shied away from radio's inability to display 
the very images that constituted the film—this intermedial gap. Instead, 
they highlighted the gap by describing the images cast onscreen in great 
detail. As he narrated the film shorts and stage action, Rothafel was telling 
audiences exactly what they were missing, exactly how their experience 
differed from that of the audiences in his theater, exactly what they were 
being denied by their radio. As Evelyn Lanzius reported, Rothafel was well 
aware that film's “chief form of entertainment, the picture, cannot be 
transmitted through the air—yet—and if people who hear the Radio con- 
cert like them in Radio form they must actually attend the theater to get 
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the picture and their interest is greatly stimulated by the Radio presenta- 
tion” (Lanzius 1923: 5). Nor was it only the film images that were lever- 
aged to help audiences visualize the show, and to draw them into the 
theater. After audiences began writing to the Capitol asking for photos of 
*Roxy and his Gang," postcards featuring portraits of various performers 
in the broadcast studio were produced and mailed to fans; by June of 
1923, it was reported that thousands of requests were in the process of 
being filled (Lanzius 1923: 5). Similarly, magazine writers were invited to 
visit the studio during broadcasts, resulting in articles that Melnick argues 
offered fans yet another “visually corollary to the disembodied voices 
audiences had grown accustomed to" (Melnick 2012: 219). I would go a 
step further: the postcard portraits and colorful descriptions that the writ- 
ers penned following their visits displayed the broadcasts in other media— 
photographs and magazine articles—with yet other sets of modalities to 
compliment the radio broadcast's intimate aural characteristics. Together, 
these media enabled audiences to better imagine the sights, smells, feel, 
and energy that accompanied the voices they heard over the air. This inter- 
medial *migration" of stage and screen performance to radio, then, not 
only bolstered the stage and screen performances themselves, but also 
prompted additional intermedial layers and movement. The resultant 
intermedial web of still and moving images, print journalism and fan let- 
ters, bodies on stage, and voices over the air traversed and defined not only 
media but time and space as well. 


1.& OPERA WITH No SONG 


In the fall of 1920, Universal released the silent film Once to Every Woman, 
directed by Allen Holubar and starring Dorothy Phillips as Aurora 
Meredith, a blacksmith’s daughter who also happens to possess a beautiful 
singing voice.’ The film joined a host of other silent films that showcased 
opera. While many of these films adapted operatic narratives for the 
screen—there were silent versions of Bizet’s Carmen, Mascagni’s 
Cavalleria Rusticana, and a host of others—others actually wove opera 
into the plot.!? Once to Every Woman is lost, but period reviews and adver- 
tisements for the film and its presentation at various theaters make clear 
that Holubar's production did both. Early in the film, a wealthy patron 
discovers Aurora singing in her village church's choir and sends her to 
Europe for training, where she makes a successful operatic debut. 
Eventually, Aurora returns to the U.S. to appear as Marguerite in Charles 
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Gounod’s opera Faust, and in hopes of distancing herself from an 
unwanted Italian admirer. As Aurora appears onstage in the opera (in the 
film), however, it is revealed that her Italian admirer has also crossed the 
Atlantic, and is seated in the theater’s balcony with a gun. As she sings, he 
shoots her and then kills himself, interrupting the opera and Aurora’s 
American debut, and throwing the theater into chaos. Eventually, Aurora 
recovers physically, but the trauma of the violence leaves her unable to 
sing. Deserted by her European friends and fans, her career all but over, 
she returns to her parents’ home alone. Even here, she finds no solace: her 
mother becomes ill and dies shortly after her homecoming. In a melodra- 
matic twist, however, this second trauma restores her singing voice: in her 
mourning, Aurora begins to sing again. 

Numerous scholars have theorized the popularity of filmic adaptations 
of opera in the silent era: opera plots were familiar, compelling, and easily 
adaptable to the screen, and opera carried the sort of artistic value and 
cultural prestige that the early film industry sought. Moreover, operatic 
adaptations relieved theater musicians and music directors of the time- 
consuming work of composing a musical score or compiling cue sheets 
from an endless selection of existing music. Instead, they could cobble 
together instrumental versions of familiar arias, choruses, and incidental 
music from a single opera score. For audience members familiar with the 
adapted operas, the resultant medleys were enjoyable, even when rendered 
solely by theater pianists, organists, or instrumental ensembles. Yet these 
sorts of films—particularly those like Once to Every Woman, which actually 
showcased opera singer characters and their performances—presented a 
fascinating challenge, too. How to represent opera singing without audi- 
ble voices? Like the Lon Chaney’s silent and sound screen ventriloquism 
and Roxy’s Capitol Theater broadcasts, Once to Every Woman oftered 
audiences a representation of one medium in another medium that lacked 
the necessary modal qualities. 

Exhibitors were clearly sensitive to this modal dissonance, and some 
developed strategies to remedy it, including the introduction of live per- 
formers into the film’s presentations. When Once to Every Woman was 
screened at the Stratford Theater in Chicago in December of 1920, for 
example, the film projection was paused during the scene in which Aurora 
appears onstage as Marguerite, and a soprano dressed in a costume mod- 
eled on Aurora’s stepped into the spotlight on the Stratford stage in front 
of the screen and sang Marguerite’s “Jewel Song” from Faust. As she 
concluded, the film projection resumed and the soprano disappeared into 
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the darkness. One film trade journal reported that the effect was seamless: 
“the impression conveyed to the audience was that Miss Phillips had 
stepped out of the screen to sing the aria” (Chicago Theatre’s presenta- 
tion elaborate 1921: 367). It was also undoubtedly dramatic: while the 
filmic medium could offer audiences Aurora’s beautiful voice only as a 
mythical, unheard object, the soprano’s appearance onstage transformed 
the silenced into the sounding, the mythic into the realized. 

The Stratford staff developed an equally compelling strategy for restor- 
ing Aurora’s voice in the theater when her mother’s death restores it 
onscreen. As Aurora opens her mouth to express her grief in the film, the 
soprano, hidden offstage, began singing the popular nineteenth-century 
song “Ben Bolt,” which remembers a woman who is now buried in a 
church cemetery. Once again, the mythical voice became manifest: that 
which audiences believed to be unavailable first due to the modalities of 
the silent film, and then due to the film’s narrative, became audible. 
Additionally, this (re-)envoicing of Aurora aligned the film’s story and its 
presentation: those in the theater experienced precisely the vocal loss and 
return that was depicted onscreen.!! 

For film audiences today, who rarely experience films accompanied by 
live musicians or bookended with stage performances, these interludes 
might seem like temporal and medial interruptions. However, the review 
quoted above suggests that audiences in 1921 probably did not perceive 
these moments as ruptures at all. Rather, the soprano's first performances 
were understood as part and parcel of the film's presentation—indeed, as 
part and parcel of the film itself. Or, to apply Andy Lavender's notion of 
performer as medium from this volume, the soprano, “herself a complex 
technical medium, operates within the larger media product of the silent 
film (Lavender 2020: 117). Once to Every Woman's intermedial play with 
opera was, like the examples above, an “as-if” situation. Though momen- 
tarily masquerading as opera, the silent film never actually became opera; 
in fact, even experiencing it *as" opera required audiences to engage in a 
significant suspension of disbelief. Yet together with Stratford Theater 
staff and performers, those audience members who were willing to 
embrace the “as-if” reimagined and remade Once to Every Woman to allow 
for audible voices. In doing so, they also temporarily reimagined and 
remade the technical medium of display of silent film, enlarging the audi- 
ence's conception of its modal boundaries and qualities to more fully 
accommodate the media product. 
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1.5 SPEAKING IN ANOTHER LANGUAGE 


In the historical examples discussed above, media producers developed 
compelling reasons and strategies to represent or mediate performances 
using technical media of display that would inevitably withhold, remove, 
truncate, and radically reconfigure those performances. My final example 
is a bit different. Club Diamond is a theater piece in which creators Saori 
Tsukada and Nikki Appino (with composer Tim Fain) consciously incor- 
porate—indeed, thematize—the sort of medial withholding, truncation, 
gaps, and “as ifs" that the historical examples stumble into. A deeply com- 
plex work, Club Diamond uses intermedial layers and combinations to 
purposefully *other" the audience, confronting them with the unfamiliar 
(and at times, that which seems unintelligible) narratively, medially, and 
structurally. By doing so, the work invites audience members to explore 
the affective experience of its protagonist(s), calling upon them to traverse 
gaps, piece things together, and compensate, just as the work's 
protagonist(s) must. 

The piece opens unassumingly. Tsukada stands center stage dressed in 
modern clothes, a large film screen behind her and a podium and clothes 
rack to her left, and cues the lights and an audio recorded monologue. 
Almost immediately, identities begin to blur. Is the first-person mono- 
logue about a childhood love of Disney and drawing Tsukada's autobiog- 
raphy or that of a character she is embodying—and why is it recorded? 
Tim Fain enters with his violin, only deepening the uncertainty: is he 
accompanist or character? Is this prologue or performance? As the recorded 
monologue ends, the pleasant haze seems to clear, momentarily: Fain 
begins tuning and Tsukada puts on a costume, declaring, *My friend Tim 
and I are going to tell you a story about how I came to the United States." 
Within seconds, though, her voice echoes as the recorded monologue 
resumes, reintroducing her, or rather, reintroducing her in her newly cos- 
tumed role: that of a prominent Japanese silent film narrator, or benshi, in 
1927.” Tsukada-as-benshi takes her place behind the podium as the film 
screen flickers alive and Fain begins to accompany the images on screen, 
but again, the performers and their roles are quickly complicated. Fain 
lowers his violin mid-phrase, revealing that he is playing (is he playing?) 
over and with prerecorded layers of music, and onscreen, the film's title 
appears: it too is called Club Diamond, and it too credits Nikki Appino as 
director (in the theater production, she is credited as a creator), Saori 
Tsukada as the principal performer (here, she is credited as “Heroine,” 
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“New Girl,” and “Bartender,” while in the theater production program 
identifies her as “performer” and creator), and Tim Fain as composer, 
along with a fleet of others.'? Is the screen simply a mirror of the stage, or 
has Club Diamond migrated from one medium to another? (See Fig. 1.2.) 

As the film begins to roll, it is immediately revealed that it is neither. 
Onstage, Tsukada-as-benshi describes the film images as they flash, draw- 
ing attention to particular elements, providing sound effects to deepen 
moods and clarify actions, and offering extensive commentary on the vari- 
ous characters and their activities—including, of course, those of Tsukada- 
as-Heroine and her other screen roles. Yet he does so almost exclusively in 
Japanese, despite the fact that the work premiered in New York City for a 
predominantly English-speaking language, and has thus far been per- 
formed for audiences mainly of English speakers. Recognizable place 
names and English expressions occasionally creep into the Jenshi’s narra- 
tion, but Japanese dominates, forcing audiences into a space of disorienta- 
tion, even incomprehension. English-speaking audience members must 
draw clues from the images onscreen, the context in which the benshi 


Fig. 1.2 Benshi (Saori Tsukada) and Musician (Tim Fain) in Club Diamond. 
(Photo by Kathryn Raines, courtesy of Nikki Appino and Saori Tsukada) 
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speaks, and his expressive face and gestures. As he dramatizes the images 
that flash on the screen in front of us, the benshi (to borrow from Andrea 
Virginás 2020: 152-159) dramatizes and prolongs the act of perceiving 
and interpreting the film, guiding audience reactions before they fully 
understand what they are seeing.!* 

Shot in black and white, the film's palette, pace, and aesthetics convinc- 
ingly pass as a 1920s production. The film includes the sort of experimen- 
tal elements, visual trickery, and cinematic “attractions” common in silent 
cinema. Structured in ten chapters, the film opens in Japan, where the 
Heroine is working as a film actress. Almost immediately, she is rejected in 
favor of a nearly identical replacement. Gutted, she is crying on a bench 
when a small bird—a symbol of opportunity in Japanese culture, an inter- 
title teaches us, and a recurrent symbol in the piece—lands near her, 
inspiring her to strike out for a new life in the Hollywood. After a long 
boat journey filled with flashbacks to the Heroine's rejection, she arrives 
in New York City. A series of montages, shadowy shots, and rapid cuts 
suggest the disorientation of the newly arrived immigrant: the Heroine's 
purse is stolen and the police pay her no mind, the streets and subway 
become a maze, crowds push and jostle. Through a series of intertitles, 
New Yorkers instruct and reprimand her: “What do you want, lady?”; “I 
DON’T SPEAK CHINESE!”; “Come on, just get on the bus!” The 
English titles do little to quell the anxious, overwhelming mood created 
by the film’s quick cuts, its fast, noisy musical accompaniment, and the 
bensh’’s rapid, audible breaths and quavering, high-pitched stuttering as 
he inhabits the Heroine. As the narrative unfolds, one wonders: is this a 
version of Tsukada’s own coming-to-America story, the story she prom- 
ised to tell? Is Tsukada-as-Heroine an avatar for Tsukada herself? Or is 
such an assumption both impatient and essentialist? 

By Chapter Five, the Heroine has found her way to a hotel, but it offers 
neither relief nor a sense of safety. It is dark and as she traverses the hall- 
way, hand-drawn spider webs appear along the walls. She cries herself to 
sleep and awakens to the sound of the bird’s wings as it alights atop a copy 
of Variety, where she finds a wanted ad seeking “Oriental Girls” to work 
at Club Diamond. At the club, she is met by Tsukada-as-Bartender, bossa 
nova-inspired music, and a male boss who takes advantage of her in her 
dressing room between performances. The film quickly turns nightmarish: 
the benshi’s narration becomes increasingly frantic and Fain's ascending 
melodies become louder and stormier as our Heroine dreams a montage 
of images and moments: her boss’s cigar, his hand on her leg, toasting 
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glasses, the doorway to her hotel, static and glitches, large black birds, 
dancing parasols, more glitches, then a black screen. “Chapter Nine: 
Revolution” flashes onscreen, but the film immediately begins to decay, 
frames and sprocketed edges rushing past and a film leader’s countdown 
numbers obscuring shots of the Heroine before the image disappears 
completely and projection ceases. The final chapter promised in the film’s 
opening titles is excised: any semblance of a storyline explodes, as does the 
silent film industry and the Jenshi’s cultural status and career, as “talking 
films” revolutionize the global entertainment scene. Just as the recorded 
monologue that began the work was punctured by Tsukada’s declaration 
that she and Fain would tell us a story, and this statement was punctured 
by Tsukada’s sudden transformation into the benshi character, now the 
benshi's performance and the medium of silent film are dramatically inter- 
rupted, leaving a bewildering, gaping fissure. 

As Tsukada-as-Heroine vanishes from view, Tsukada's recorded mono- 
logue resumes. The last time the audience heard this voice, it was intro- 
ducing the benshi to us. Here, it continues narrating his story, describing 
the death of a career and lifelong love: “You want to ride it out. You try to 
ride it out. But suddenly, you become [...] ancient.” Tsukada-as- benshi 
removes his stage outfit slowly, inhabiting the rhetorical “you” of the 
monologue more fully with each movement. But the monologue pivots, 
and Tsukada’s voice describes the desperation that drove her to answer a 
newspaper ad titled “looking for girls,” recounting an experience working 
as an escort that parallels Tsukada-as-Heroine’s and further collapses the 
two characters into one, despite the century gap. 

Meanwhile, in the corner of the stage, Tsukada has transformed from 
benshi into kamishibatya, a type of itinerant street performer taken up by 
many benshi after the introduction of talkies, who entertained children 
with candy and paper plays in postwar occupied Japan.!° In white cover- 
alls, a hat, and goggles, Tsukada-as-kamishibaiya mounts a bicycle with a 
large wooden box (butai) on the back, and rides unsteadily to center stage, 
radio blaring. He parks and sets his stage, opening the butai to reveal a 
candy display and a set of illustrated storyboards displayed in a frame, and 
then begins his performance. An informal street act from the back of a 
bicycle for Japanese and American expat children, the kamishibaiya’s per- 
formance lacks the polish and ceremony of the benshi’s. Yet the audience 
is encouraged to understand the ragtag street performer as the benshi rein- 
vented. Tsukada-as-kamishibaiya retells the same story that Tsukada-as- 
benshi did earlier, narrating the heartbroken Japanese film star's trip to 
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New York, her frantic navigation of the unfamiliar city, and the rundown 
hotel and club where she finds herself living and working. Though the 
kamishibaiya uses a few more English words and phrases than he had as 
benshi, the narration remains largely in Japanese, and is performed with 
the same dreaminess and deep emotional investment. Even the images 
featured on the storyboards harken back to the silent film: several are exact 
replicas of illustrations from the film. This mirroring of kamishibaiya and 
benshi hints at the continuity of another pair that, though separated by 
time, are increasingly difficult to tease apart: our film heroine and the 
Tsukada character who emerges in the recorded monologue. 

The paper play ends as abruptly, and as traumatically, as the film—and 
at exactly the same point in the narrative. The Heroine’s work at the club 
is introduced with a bright red card with swoops of yellow that resemble 
stage curtains and the same bossa nova tune that accompanied the club 
scenes in the film, and suddenly, the kamishibaiya abandons his narration 
to move about the stage, stomping his feet and beating a shaker against his 
palm percussively. As narrator becomes musician, musician becomes 
dancer: Fain moves increasingly extravagantly to his own violin music and 
the kamishibaiya’s rhythms. It is a musical duet and pas de deux in one, the 
violinist seemingly drawn into the kamishibaiya’s postwar Japanese world, 
into his paper play, into Club Diamond. The musician and kamishibaiya 
whirl in synchronization, crouching and then unfurling their bodies to 
stand upright only to bend again, more quickly, more frantically, as the 
music grows increasingly disjoined and then suddenly, stops. The two pull 
apart and the kamishibaiya resumes his narration, quiet and still. Then, a 
flourish: he tosses a handful of confetti and, as Fain begins a pentatonic 
pizzicato passage, removes the red storyboard to reveal a green card cov- 
ered with white confetti. Instead of quickly hiding the previous board, 
though, he gracefully flips it overhead, revealing a muted image of the 
rising sun flag before depositing the board behind the others. It is a fleet- 
ing but potent symbol that hints at the kamishibaiya’s worldview and ori- 
ents the entire scene in time and space, differentiating this telling of the 
Club Diamond tale from the earlier silent film. Tension grows as the 
kamishibaiya continues on, mimicking the heroine’s panicked high voice 
and beating a drum attached to his bicycle with increasing power until he 
bends from the waist, gasping and crying—an embodied rendition of the 
heroine’s bad dream. He straightens, covers the storyboards, and abruptly 
dismisses the crowd: “The end. Everybody go home.” 
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Unexpectedly, though, the kamishibaiya does not exit. Instead, accom- 
panied by a haunting ascending violin melody, he wheels his bicycle to the 
right side of the darkened stage, where he pulls open the reverse side of 
the butai to reveal a built-in film projector. Squatting next to it, he extends 
the projector’s lens forward, flips on the lamp, and begins rotating the 
bicycle’s pedals to power the projector. An intertitle identical to those in 
the Club Diamond silent film flashes onscreen, introducing the final, 
unseen segment of the film: “Chapter Ten.” A Universal Studios 
International News Reel titled “Oriental Cutie makes it big in Hollywood!” 
begins to roll, presenting Tsukada-as-Heroine blowing kisses to her ador- 
ing fans. The newsreel is a talkie, with a jaunty orchestral march accompa- 
nying the opening action and then voiceover narration regaling us with 
tales of this “new born” film star: the final chapter has finally arrived, and 
it has arrived in sound. The film captures a final shot of Tsukada-as- 
Heroine outdoors, dressed in white; the camera lingers on her face, and 
then a “The End” title flashes onscreen, with a subtitle added a few sec- 
onds later: “Made in the USA.” Tsukada-as-kamishibaiya slows the bike 
pedals as the image flickers out. He rises, removes his goggles and hat; 
suddenly, it is Tsukada who stands before us blinking, then licks her lips 
and begins a live monologue about the drawings that she would make and 
fax to her parents in Japan to show them her new New York life, and her 
father’s advice that she keep talking, even imperfectly, because Americans 
do not like silence. Delivered quietly and haltingly, the monologue stitches 
together the childhood reminiscences from the first recorded monologue, 
her fraught adjustment to New York City detailed in the second recorded 
monologue, the kamishibai drawings, the fate of the benshi in our culture 
of sound and talk, and the linguistic and social gulfs between Japanese and 
U.S. culture that the benshi, Tsukada-as-Heroine, the kamishibaiya enter- 
taining postwar children, and Tsukada herself all face. In this final, sub- 
dued moment, Tsukada’s various characters collapse into one another 
even more profoundly, across time and space, gender and career, identity. 
Onstage, Tsukada points upward, cueing the lights. 

Brilliantly coherent as it moves from stage to screen to paper play and 
back, Club Diamond nonetheless generates the same sorts of medial and 
narrative gaps, fissures, and disorientation as my earlier examples. In addi- 
tion to the overlapping, intertwined characters that lead audiences to con- 
stantly question who is speaking, for whom, and whose stories we are 
witnessing, Club Diamond positions English-speaking audiences as lin- 
guistic “others,” thematizing the disorientation and partial understanding 
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so familiar to immigrants and travelers.'° Creating the piece, Appino and 
Tsukada meticulously constructed the benshi and kamishibaiya’s speeches 
such that one word, perfectly hit, might carry an entire paragraph, and 
generated visual cues to highlight the importance of visuality and looking 
when navigating linguistically inaccessible spaces. Nonetheless, these sec- 
tions withhold as much as they reveal, prompting viewers to constantly 
question what and whether they understand, and to settle uneasily into a 
space of not quite knowing. 

For most audience members, the figures of the benshi and kamishibaiya 
are as unfamiliar as the language of their narration. Indeed, the benshi 
tradition itself is a uniquely Japanese practice of which relatively few 
Americans are aware, and even fewer are familiar. For that matter, though 
silent film has its enthusiasts, it likely seems quaint and distant to modern 
audiences. Kamishibai performances are even more mysterious in an 
American context. Despite the ubiquity of these performers in postwar 
Japan, the medium has no real American corollary. While the kamishibaiya 
and benshi may offer historical, geographical, and political clues to some 
audience members, then, many more are likely to experience them as lack- 
ing contextual qualification, to borrow a useful term from Ellestróm 
(2020: 60-64): removed from their historical, social, and cultural contexts 
and represented in a contemporary work of the theater, the benshi and 
kamishibaiya lose a bit of their intelligibility. Yet the use of contextually 
unqualified media is precisely what enables Club Diamond to generate 
gaps in which audience members must flounder, compensate, piece 
together, and reorient. 

The gaps and uncertainty built into Club Diamond come with costs. 
Linguistic unfamiliarity makes it easy to miss narrative action. Audiences 
less familiar with twentieth-century politics, media history, and perfor- 
mance practices can easily find misunderstand when or where various sec- 
tions of the work are set. Important details can slip by without registering, 
particularly while audience members work to piece other details together 
in their minds. Even as a music and media scholar who studies silent film 
and performance, I overlooked a number of compelling moments in my 
first several interactions with the work—and none of them were particu- 
larly nuanced or veiled. In a Skype conversation with me (17 July 2019), 
Tsukada noted that younger audience members, in particular, seem to 
miss visual and historical cues, in large part because they do not have the 
historical framework through which to “read” the work’s characters and 
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action. As a result, she and Appino continue to consider if and how they 
should make certain elements—such as the political and cultural context 
in which the kamishibaiya performs, and the political overtones that make 
their way into his performance when, for example, he displays the rising 
sun on the reverse of a storyboard—more legible to audiences who lack 
the historical understanding to situate him and his actions. 

The pleasure—and ultimately the legibility—of Club Diamond depends 
in large part on the intermedial web it constructs. Tsukada's immigration 
story is told in a series of live and recorded monologues; a second, slightly 
abstracted version of that story is told on film and in a paper play. Each 
time the narrative is cut off, a medial transposition is forced; each medial 
transposition offers audiences a new set of modalities through which to 
experience and interpret this story. Tsukada’s second monologue, for 
instance, translates moving images into spoken words, adding precision 
and detail. Similarly, the simple images of Tsukada-as-kamishibaiya’s paper 
play highlight and distill themes easily lost in the silent film’s elaborate 
moving images or the pace of the monologues. With each medial transpo- 
sition, too, layers of significance accrue around the work’s recurrent sym- 
bols, and the audience is resituated in space and time, moving from the 
1920s and 1930s to the late 1940s and 1950s and on to the present, and 
between Japan and New York. 

Club Diamond showcases the continuity and synergies not only of the 
narratives told within these various forms and genres, but also of the forms 
and genres themselves. The piece narrates the evolution of silent film into 
sound film, and more subtly suggests the emergence of kamishibai from 
the ashes of the silent film industry. By virtue of casting and narrative, the 
benshi and kamishibaiya are also inseparable from the silent film starlet, the 
contemporary immigrant actress and Tsukada herself. Moreover, aesthet- 
ics and modal qualities associated with one medium are frequently 
deployed or at least invoked in other media in the piece. With the repeated 
interruptions and truncation of the narrative, for example, Appino and 
Tsukada invoke a characteristic of kamishibai paper plays: kamishibai art- 
ists regularly omitted the final scenes of their stories in order to encourage 
repeat “customers.” The technique is equally effective here: the refusal of 
narrative closure builds anticipation such that the kamishibaiya’s quiet, 
private viewing of “Chapter 10” becomes a strikingly beautiful and 
profound moment, a fulfillment of the work’s unspoken promise to its 
audience. Similarly, for English-speaking audiences, the benshi and 
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kamishibaiya’s Japanese narrations come to function much like silent film 
musical accompaniment (including Fain’s within the work): they establish 
mood, guide emotional registrar, and offer hints about action more than 
they legibly narrate the action of the film or paper play. Indeed, it is fre- 
quently difficult to tell which sound effects are generated by the benshi and 
kamishibaiya, and which come from Fain’s violin and prerecorded mate- 
rial; when Fain and Tsukada-as-kamishibaiya begin to move and make 
music in synchronization, their sonic, dancing bodies transcend simple 
boundaries between music, dance, and narrative. The creators of Club 
Diamond use intermediality to generate in audiences precisely the feeling 
of disorientation that Tsukada and her alter egos describe, enact, and 
embody. Yet the gaps, fissures, and sensations that result also reveal much 
more: the limits and possibilities of performance, the intimate connections 
between seemingly disparate media, and the aesthetic and narrative prom- 
ise of medial transposition and borrowing. 


1.6 | CONCLUSION 


Club Diamond, Roxy’s broadcasts, Once to Every Woman, and The Unholy 
Three do not convey meanings and ideas as expected. As such, they remind 
us that mediation is not only about seamless and easy transmission of sen- 
sations and ideas across space, time, and bodies. Mediation can also 
prompt experimentation within a media product and/or a technical 
medium of display, as The Unholy Three does, or make tangible the unique 
possibilities and intermedial potential of multiple media, as in Roxy’s 
Capitol Theater broadcasts. Mediation can push producers and perceivers 
to reimagine and differently experience medial boundaries, as in the 
Stratford’s presentation of Once to Every Woman. Finally, mediation can 
purposefully and productively refuse smooth transmission in order to gen- 
erate particular perceptual effects and bodily affects, as in Club Diamond. 
In each of the examples described above, and innumerable others, inter- 
mediality foregrounds various media’s modal limits and mismatches, their 
expressive incompatibilities and shortcomings. At the same time, interme- 
diality opens up new expressive fields and modes of representation, and 
stages unexpected perceptual possibilities. In his essay on media and the- 
ater, Peter Boenisch notes that intermediality is not just about transmis- 
sion of messages, but also about activating audiences, inviting them into 
complex imagined worlds; it is as much a perceptual effect generated in 
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performance as an inherent quality (Boenisch 2006: 114-115). Perhaps, 
then, intermediality is most potently generated in performances that chal- 
lenge—and at times confound—the audience’s expectations and under- 
standings of media. 


NOTES 


l. A variety of different intermedial “techniques” are on display in my exam- 
ples. Some engage in medial transposition, moving and representing narra- 
tives and characters from one medium to another. Others borrow aesthetic 
conventions associated with one medium and deploy them in another. Still 
others recombine media in unexpected ways, or represent a series of media 
in another medium, one after the next. For more on these various “types” 
of intermediality, see, for example, Rajewsky (2005) and Chapple and 
Kattenbelt (2006). 

2. Boenisch argues that intermedial performances can be profoundly unset- 
tling for audiences, creating “effects of alienation and dys-referential un- 
realities" (2006: 115). 

3. The 1925 version of The Unholy Three survives, as does the 1930 sound 
version discussed later in this section of the essay. Both are available com- 
mercially. See The Unholy Three (1925c). Directed by Tod Browning 
(2010; Warner Home Video), DVD and The Unholy Three (19302). 
Directed by Jack Conway (2010; Warner Archives), DVD. 

4. Among the numerous scholars who have made this claim is Rick Altman, 
who in his essay *Moving lips" argues that the illusion that film soundtracks 
are redundant and subservient to the image actually serves to constitute 
filmic unity. Film so prominently displays the speaking character precisely 
because seeing their moving lips helps to “transfer the origin of the words, 
as perceived by the spectator/auditor, from sound ‘track’ and loudspeaker 
to a character within the film's diegesis," disguising the film's construction 
and technologies (Altman 1980: 67, 69). 

5. The speech bubble text in particular successfully remediates another popu- 
lar early twentieth-century medium with which most filmgoers in 1925 
would have been familiar: the comic strip. 

6. Rothafel was among the earliest to so fully incorporate radio into his the- 
ater activities, but other exhibitors and film studios began experimenting 
with radio around the same time, and by 1924, many of Broadway's large 
film theaters, and those located in other major American cities, were also 
broadcasting their stage shows. See Douglas Gomery (2005: 16) for addi- 
tional examples of silent film/radio experiments. 
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. For more on remediation, see Chiel Kattenbelt, “Intermediality: A redefi- 
nition of media and a resensibilization of perception”, keynote lecture at 
Intermediality: Performance and Pedagogy conference at Sheffield 
University, 15 March 2007; Bolter and Grusin (1999: 56); Nikunen 
(2007: 113); Auslander (2003: 24). 

. This act of virtually recasting a media object in its original technical 
medium of display, or even immersing oneself in a narrative unbounded by 
a technical medium of display altogether involves the cross-modal cogni- 
tive capacities that Lars Ellestróm discusses in his “The modalities of media 
II" (2020: 68-69). 

. I discuss this film and its presentation at greater length in Simonson (2019). 

. For more on the relationship between silent film and opera, see Grover- 

Friedlander (1999, 2002), Stern (2002), and Simonson (2013). 

In addition to allowing audiences to hear a long-lost voice, this moment 

also enabled them to hear the specific song for which many were waiting. 

When the film was previewed at the Hotel Astor, a soprano was hired to 

sing “Ben Bolt" both before the screening began and when Aurora sings 

at her mother's bedside. The stunt was so well received that Universal 
launched a “song tie-up” campaign: they issued a special ad book contain- 
ing piano-vocal sheet music and advised exhibitors to ask their local music 
stores to stock the sheet music, hold a weekly giveaway of one copy, and to 
feature in their window displays to help promote the film. If done right, 

Universal told exhibitors, audiences “will all want a copy of this song— 

they'll all be singing it—they’ll be reading it. And when you show the 

picture, they'll be jamming your house during the entire run." See 

“Universal arranges song tie-up” (1920: 4446) and “Special Universal ad 

booklet for Once to Every Woman (1920: 18). 

Benshi were considered crucial to film exhibition in Japan during the silent 

era, and often achieved levels of fame that surpassed that of film stars. For 

additional information about the practice, see, for example, Gerow (2010). 

Tsukada's performance of these multiple, intertwined characters might 

usefully be analyzed through the multimodal performance framework that 

Andy Lavender proposes in this volume, based on a transposition of 

Ellestróm's media modalities (Lavender 2020). 

The movie screen in C/ub Diamond functions similarly to (and might be 

considered) what Andrea Virginás labels a diegetic screen in her essay in 

this volume. As she notes, these screens are watched by diegetic characters 

(here, the benshi), directing our attention to — Ellestróm's 

intracommunicational domain and heightening the act of perception that 

precedes interpretation (Virginás 2020: 152-159). 

For more about kamishibai and kamishibai artists, see Orbaugh (2015) 

and McGowan (2015). 
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16. To be clear, I am not suggesting that media require language to communi- 
cate; I agree with Ellestróm's identification of the problematic nature of 
such assumptions in his essay in this volume (Ellestróm 2020: 6). Rather, I 
am pointing out that it is striking that C/ub Diamond incorporates lan- 
guage as much to obscure as to transmit. Indeed, C/ub Diamond displays 
precisely media's ability to communicate despite language, as well as 
through language. 
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Transferring Handmaids: Iconography, 
Adaptation, and Intermediality 
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2.1 INTRODUCTION 


All around us, depictions of red-cloaked, white-bonneted female silhou- 
ettes proliferate. An image of the forty red topiaries that lined the east 
colonnade of the White House in the 2018 Christmas season, edited to 
appear topped with white “Handmaid” bonnets, went viral on Twitter in 
November 2018. A June 9, 2019 Associated Press article entitled 
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"Scientists feel chill of crackdown on research" is accompanied by a May 
23, 2017 photograph of women donning red cloaks and white bonnets at 
a protest against Texas SB8, a bill which bans second trimester abortion, 
and requires that fetal remains be buried or cremated (Neergaard et al. 
2019). A lingerie website stocked a “Brave Red Maiden” costume “featur- 
ing a red mini dress, a matching cloak with attached hood, and a white 
bonnet headpiece” (Gupta 2018).! 

While many viewers recognize this iconography from Margaret 
Atwood's novel, The Handmaid's Tale (1985) or the Hulu series (2017-) 
of the same name, a recognition of the source does not explain how the 
red cloak and white bonnet have come to signify in contexts as diverse as 
a topiary display, threats to scientific research, and lingerie. In the thirty- 
five years since its publication, The Handmaid’s Tale has been adapted 
numerous times in a range of media. Reasons for its longevity are many: 
the novel’s popularity and status as a classic work in dystopian literature 
make it ripe for adaptation, as does the fact that it has already been adapted, 
as works that have been adapted one or more times seem to be prime can- 
didates for future adaptation. Another, perhaps more telling reason behind 
the continued adaptation is a general feeling that the novel’s themes reflect 
the political dis-ease of a particular moment. Conversations related to any 
adaptation of The Handmaia’s Tale, regardless of the decade, are likely to 
mention the contemporary political climate in the United States, and else- 
where.’ This sense of sustained relevance means that The Handmaid’s Tale 
is almost always read through and against broad contextual lenses at the 
same time that the novel becomes the point of reference for that context. 

This essay examines the intermedial transfer of Handmaid iconography 
across platforms and contexts, and the mechanisms that facilitate such 
movement. I begin with a consideration of the intermedial network estab- 
lished within Atwood's novel to show that, even prior to adaptation, the 
Handmaid is understood as a product of intermedial transfer. I next survey 
the movement of Handmaid iconography in a variety of seemingly static 
print-based and dynamic motion-based media, and then turn to articula- 
tions of Handmaid iconography in more generalized spheres. The image 
of the Handmaid transfers through processes of adaptation that interpret 
visual markers in distinct modalities, each of which emphasizes particular 
traits or characteristics over others. The emphasis or disclosure that char- 
acterizes each iteration is accompanied by concealment; that is, as an adap- 
tation. foregrounds a particular modality, it simultaneously represses 
another. In the context of The Handmaid’s Tale, the tension between 
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disclosing and concealing, between visibility and invisibility operates the- 
matically and in terms of its foregrounded media. 


2.2  INTERMEDIALITY, ICONOGRAPHY, AND TRANSFER 


There are many ways to think about intermediality, iconography, and 
transfer. My own view of intermediality aligns with that described by Irina 
O. Rajewsky in her contribution to the essay collection Media Borders, 
Multimodality, and Intermodality. “Generally speaking, and according to 
common understanding, 'intermediality refers to relations between 
media, to medial interactions and interferences. Hence, ‘intermediality’ 
can be said to serve first and foremost as a flexible generic term ‘that can 
be applied, in a broad sense, to any phenomenon involving more than one 
medium’ and thus any phenomenon that—as indicated by the prefix 
inter—in some way takes place between media. Accordingly, the crossing 
of media borders has been defined as a founding category of intermedial- 
ity” (Rajewsky 2010: 51—52).? Rajewsky proposes viewing borders as sites 
of possibility, “as enabling structures, as spaces in which we can test and 
experiment with a plethora of different strategies” (65). The broadness of 
this conceptualization, which I find welcome, may frustrate others who 
approach intermediality more narrowly or who would prefer a more cir- 
cumscribed set of terms that delineate media borders and avenues of 
“crossing.” Yet, as Rajewsky, Lars Ellestróm, and other contributors to 
this volume illuminate, such borders, while useful theoretically, are always 
constructed and perceptual. That is, no material ‘border” exists between, 
say, the animated and live-action segments of a particular film, yet audi- 
ences perceive aesthetic differences, and articulate that difference in terms 
of juncture and border crossing. 

Throughout this essay, I refer to the movement or transfer of iconogra- 
phy from one media product to another. Here, I use “iconography” to 
refer to a dynamic and fluid collection of visual symbolic information that 
signifies to perceivers a distinct “message”—be that message a particular 
cultural object or other form of communication. By “transfer,” I mean the 
generalized perception that messaging associated with one place “moves” 
to another.* That which is transferred may be iconography or theme or 
tone or narrative or character or compositional principle, or any number 
of identifiable aspects. The perception that transfer has occurred does not 
mean that the thing no longer exists in the previous location (or previous 
locations before that). My concept of transfer is informed by a model of 
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communication put forward by Ellestróm in “The modalities of media II: 
An expanded model for understanding intermedial relations” (2020). 
Ellestróm identifies “three indispensable and interconnected entities” nec- 
essary for communication to occur between a “producer’s mind" and a 
“perceiver’s mind” (2020: 10, 12): “1. Something being transferred”; “2. 
Two separate places between which the transfer occurs”; “3. An interme- 
diate stage that makes the transfer possible” (2020: 10). 

All media products are communicated by way of their materiality, and, 
likewise, the perception of transfer manifests in materiality. Ellestróm's 
model identifies “four kinds of basic traits,” or “media modalities”; these 
include the material, spatiotemporal, sensorial, and semiotic modalities. 
All media products are realized through some materiality (e.g., flat, raised, 
solid), and experienced spatiotemporally (e.g., two-dimensional, three- 
dimensional, sequential, non-sequential) and sensorially (e.g., audial, 
visual). The semiotic modality is *derived from" the other three modali- 
ties. That is, the perceiver takes in material, spatiotemporal, and sensorial 
information and interprets that information as signifying (2020: 20). 
Important to the process of communication is the manner by which per- 
ceivers understand their particular experience of a given message as com- 
pared to and within the context of previous perceptions and 
communications—what Charles S. Peirce called “collateral experience” 
(Ellestróm 2020: 26-27). In the course of examining the movement of 
Handmaid iconography, I show how each adaptation foregrounds certain 
modalities to lead perceivers toward particular interpretations of the com- 
munication transfer. 


2.3 MEDIATION IN THE HANDMAID’S TALE (1985) 


“What’s fascinating about the legacy of The Handmaid’s Tale,” Natalie 
Zutter writes, “is how it’s spread to almost every medium: reimagined on 
stage and screen, buzzing on the airwaves and between your ears, inked 
earnestly onto skin and snarkily onto protest signs, embodied in real bod- 
ies through viral marketing and political action. [...] you can see Offred’s 
story from tech conferences to the Senate floor” (2017). Zutter’s verb 
choices—“reimagined,” “buzzing,” “inked,” “embodied”—identify mul- 
tiple avenues of mediation that point both to communication transfer 
(e.g., how particular messages are produced, transferred, and perceived) 
and medial transfer (e.g., how The Handmaid’s Tale is adapted from novel 
to television, radio, and commercial and political arenas). This aspect of 
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The Handmaid’s Tales legacy is not as surprising as it might seem at first 
glance. Atwood incorporates into the novel numerous medial references, 
both within the “tale” and as an organizing tool for the narrative conceit. 
Such moments indicate a keen awareness of how media and mediation 
function in concert to construct and convey meaning, as well as how shifts 
in media impact perception. The “Historical Notes” section, which 
appears at the end of the “tale” proper, purports to provide “a partial 
transcript” of a “live” event—the 2195 conference proceedings of the 
Twelfth Symposium on Gileadean Studies, in which Professor James Darcy 
Pieixoto conveys orally the context of the transcribed “The Handmaid’s 
Tale” in a talk entitled, “Problems of Authentication in Reference to The 
Handmaid’s Tale” (Atwood 1985: 312). In this turn, readers realize the 
conceit: the text that they thought they had been reading as a fictional 
novel is presented as both an excerpt from conference proceedings as well 
as the supplement on which those proceedings are based. 

Professor Pieixoto's comments alert his audience to the elaborate net- 
work of transmedial transfer that has informed his presentation. The 
Handmaid's *tale? had been invisible, recorded onto some thirty cassette 
tapes over existent musical recordings. At the time of the discovery, cas- 
sette tape technology had long been obsolete. However, Professor Pieixoto 
and his colleague, Professor Wade, have “with the aid of [their] excellent 
resident antiquarian technician, reconstructed a machine capable of play- 
ing such tapes" (313). For Pieixoto and Wade, the concern in transferring 
an oral account to audio recording and then to transcript and then to oral 
presentation rests in the authenticity of the medial relationships and cross- 
ings. Pieixoto states that ^we were assured by the experts who examined 
[the tapes] that the physical objects themselves are genuine. Certainly, the 
recording itself, that is, the superimposition of voice upon music tape, 
could not have been done within the past hundred and fifty years" (315). 
For these scholars, authenticity of medium and authenticity of narrative 
are yoked—the narrative's authenticity comes into being through the pro- 
cesses of adaptation and transmediation that allow the narrative to be 
communicated. Visibility and invisibility factor into their establishment 
and assessment of authenticity in that the superimposition of one media 
type (oral speech) on another (music) results in the necessary concealment 
of one message to reveal another. 

Atwood prepares her readers for such challenges of mediation within 
the “tale” proper. We know, for example, that the narrator's job “in the 
time before,” prior to the instatement of the government of Gilead, was in 
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a library, *transferring books to computer discs, to cut down on storage 
space and replacement costs, they said.” The narrator relates that, “After 
the books were transferred they were supposed to go to the shredder, but 
sometimes I took them home with me. I liked the feel of them, and the 
look" (182). Here, the act of transfer renders one technology (print, in 
this case) obsolete; once content is transferred, the form is destroyed. The 
narrator, however, appears less interested in the content or semiotic value 
of the media product, and, instead, fetishizes its sensorial and material 
qualities—she *like[s] the feel of them, and the look." The narrator serves 
as the intermediate stage in the communication process—she facilitates 
the movement of communication from one place to another, and can also 
redirect that communication. In bringing a book home, she initiates a new 
communication and expands the intermedial network. The hypothetical 
book now exists in the library's records of material objects, as content 
transferred to disc, and as an aesthetic object in the narrator's home. The 
book has not abandoned any of these communication points, but exists in 
all simultaneously. 

While the verb “transfer” suggests a seamless activity, a “move from 
one place to another," distinctions in definitions of transfer highlight a 
process that is considerably more fraught. Transfer means to *copy (data, 
music, etc.) from one medium or device to another." Transfer can also 
mean, “make over the possession of (property, a right, or a responsibility) 
to someone else" (Transfer 2019). Each of these definitions signifies dif- 
ferently in The Handmaid^s Tale, particularly as the narrator herself experi- 
ences "*transfer"—from freedom to bondage, from the Rachel and Leah 
Re-education Center (Red Center) to an assignment as a Handmaid, from 
the home of one Commander to that of another, from knowing to 
unknowing. Each of these definitions signifies differently in The 
Handmaid’s Tales adaptation history, as well. Much in the way that the 
transfer of speech to cassette and of recording to paper involves shifts in 
medial interface and reframing of what is seen and emphasized, transfer of, 
say, a novel from its physical, dimensional book materiality to a physical, 
dimensional disc materiality involves a reframing—a reframing of how that 
novel is read (paper vs. screen, page turn vs. scroll) and understood as a 
sensory object (smell, touch)—and a tacit understanding that all materiali- 
ties are prone to obsolescence. 
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2.4 THE HANDMAID: ESTABLISHING A NETWORK 
OF ICONOGRAPHY 


Early in the novel, Atwood’s narrator describes the features of her 
Handmaid uniform: 


I get up out of the chair, advance my feet into the sunlight, in their red 
shoes, flat-heeled to save the spine and not for dancing. The red gloves are 
lying on the bed. I pick them up, pull them onto my hands, finger by finger. 
Everything except the wings around my face is red: the colour of blood, 
which defines us. The skirt is ankle-length, full, gathered to a flat yoke that 
extends over the breasts, the sleeves are full. The white wings too are pre- 
scribed issue; they are to keep us from seeing, but also from being seen. 
(1985: 18) 


This description demonstrates to readers the boundaries between 
*Handmaid" and “not Handmaid,” as well as that between “individual” 
and *Handmaid" through statements that establish both the narrator's 
awareness of her subject position (i.e., “I”) and the object position of 
Handmaid (e.g., “their red shoes,” “the breasts"). The specifics of the 
uniform cannot be separated from its task of creating distanciation between 
self and role, and of rendering Handmaids both highly visible and invisi- 
ble. As the narrator, called Offred within Gilead, learns during her reedu- 
cation process at the Red Center, *[m]odesty is invisibility [...].To be 
seen—to be seen—is to be [...] penetrated” (39). The paradox in this 
statement is that the Handmaid's job is to be penetrated, monthly, by a 
Gilead Commander, in the context ofthe government-ordained Ceremony. 
The danger is in being *seen" as an individual, in a perceiver acknowledg- 
ing the boundary between Handmaid and individual. “Modesty,” as 
defined within Gilead, protects that boundary. The uniform renders all 
Handmaids more or less anonymous visual equivalents (as reinforced in 
the narrator's blurring of plural/singular usage in describing her move- 
ments with a fellow Handmaid: “Doubled, I walk the street” [33]). The 
red uniform transforms the women who wear it into symbols, both of life- 
bringing possibilities, and of sexual threat and promiscuity. In explaining 
a character's coldness toward her, the narrator explains, *the frown isn't 
personal: it's the red dress she disapproves of, and what it stands for" (19). 
There are certain exceptions to the Handmaids’ anonymity, of course. 
Pregnant Handmaids are made visible, individualized by their swollen 
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bellies, which distend the front of the red dress, altering its shape. 
Handmaids are also visible to each other: they see each other as distinct 
women wearing the uniform, rather than as “Handmaid.” Such instances 
of recognition draw attention to the constructed nature of these individual 
and thematic borders. 

Atwood's descriptions of the uniform are not quite ekphrastic, but offer 
enough detail to convey to readers a clear image of the uniform, while 
remaining general enough to invite flexibility in interpretation. Fred 
Marcellino’s jacket illustration for the first American edition of The 
Handmaia’s Tale, the 1986 Jonathan Cape edition, marks the first transfer 
of the iconic Handmaid uniform from written, verbal description to a two- 
dimensional visual illustration, materialized through color, shape, line, 
typography, and other attributes of visual design. Marcellino’s illustration 
depicts two Handmaids walking along a tall, gray brick wall. The emphasis 
is on the vivid red cloak and distinctive winged bonnet. The image’s 
“bird’s eye” perspective and juxtaposition of the Handmaids against the 
insurmountable wall evoke the dystopic society of the city of Gilead imag- 
ined in Atwood's novel, as does the anonymity of the figures. Even here, 
though, the image explores the boundary between the individual and the 
collective. The Handmaids are slightly staggered in the illustration, one 
moving away from the other, who has turned to look behind her—a ges- 
ture that imbues the image with a sense of expectation, and suggests sub- 
tly differences in motivations of individual Handmaids. The particular 
moment singled out by Marcellino is not based on a specific moment from 
Atwood's novel, but appears to be a generalized summation of the novel’s 
numerous scenes of walking and looking. The particular shape and color 
saturation of the Handmaids’ cloaks, as well as the upturn of the wings on 
the white headdress align with the narrator’s description, but are also the 
product of Marcellino’s transfer of visual verbal description into visual 
iconic signifiers. 

Marcellino’s interpretation initiates a method of visually articulating 
the Handmaid in book jacket design. A tour of The Handmaid’s Tale book 
covers shows Marcellino’s iconography transferring from edition to edi- 
tion through a repetition of basic thematic and visual coding. For exam- 
ple, a 1998 McClelland and Stewart edition likewise features a tall gray 
brick wall, curving at roughly the same arc as that in Marcellino’s image, 
and adopts also a bird’s eye perspective, but depicts the Handmaid 
abstractly. The center of the image features a piece of red cloth blowing in 
the wind, bent into the shape of a Handmaid’s cloak. The effect, then, is 
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that of an escape, of the Handmaid flying up and over the wall. Other 
examples include the 1995 Virago Press cover, which features two 
Handmaids walking along a low wall and row of tall, spike-formed topiar- 
ies. The women are “doubled,” to borrow Atwood's narrator's descrip- 
tion, and walk with heads bent and baskets clasped. 

As the iconography of the Handmaid transfers from edition to edition, 
the effort to imbue it with secondary symbolism becomes evident, par- 
ticularly symbolism that underscores an aspect of the Handmaids’ oppres- 
sion (such as an inability to speak, as is emphasized in the cover of the 
1986 Seal Books edition, which depicts a Handmaid with her lips bound 
by metal rings) or that suggests freedom. For example, the impressionistic 
quality of a 1996 Vintage, alternative paperback edition cover obscures 
the Handmaid's identity, while the saturation of red heightens the figure's 
visibility. The warm color palette coupled with the modeling technique 
suggest a melting of the Handmaid's face—alluding both to her anonym- 
ity (one Handmaid *melts" into the next), as well as the violence suffered 
by Handmaids at the will of the Aunts and others. A Vintage cover for a 
2007 edition depicts the Handmaid in silhouette, kneeling before a fruit 
tree, staring upward beseechingly. Her gloved hands contort into a shape 
reminiscent of shadow-cast birds, and two white birds fly from her torso— 
visual extensions of the white bonnet. This cover incorporates into its 
design a crescent moon shape associated with the Red Center as well as the 
biohazard symbol, thus alluding to both the Handmaids’ role in propaga- 
tion and the punishment for women who “fail” at that role. The tree is 
heavy with pome fruit, underscoring the Handmaid's reproductive poten- 
tial, though three pieces have fallen to the ground, suggesting unrealized 
opportunities. 

While several covers focus on the singular Handmaid, others strive to 
represent graphically the uniformity of the Handmaids. The 2010 Vintage 
Classics edition, for example, presents a row of faceless, identical screen- 
printed Handmaids, reminiscent of mannequins or robots, that rare books’ 
specialist Rebecca Romney compares to Doctor Who’s Cybermen (2017). 
The 2017 Vintage edition designed by Suzanne Dean, featuring an illus- 
tration by Noma Bar, represents the Handmaid graphically on the front 
cover, signaling the Handmaid through the outline of a full red dress and 
white bonnet, and, again, on the back cover, through a keyhole-shaped 
representation of a Handmaid, which is then repeated in a pattern in the 
edition's front and end papers. A 2015 translated Georgian edition like- 
wise utilizes the familiar graphic signification of the Handmaid via the 
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silhouette of the red dress and white bonnet, and incorporates doubled 
figures against a gray wall. However, the doubling is ofa Handmaid and a 
Wife, rather than two Handmaids. The silhouette of the wife appears 
behind the Handmaid, signified by blue dress and short veil. She holds a 
baby, coded red, that remains attached to the Handmaid via the umbilical 
cord, also red. The brick wall is incorporated as the shadow cast by the 
Handmaid, and her figure is visually segmented, divided at the chest and 
waist and mid-skirt. Through visual echoing, this image highlights the 
distinct functions of the Handmaids and the wives, while simultaneously 
alluding to the uniformity of those roles. 

If we think of a book cover as offering a visual shorthand of a book's 
most essential information, the continued repetition and reinterpretation 
of the red cloak and white bonnet reinforces these features as essential 
signifiers of both the figure of Handmaid and The Handmaid’s Tale. As I 
have discussed elsewhere, the repetition of specific iconography through 
successive adaptations across multiple media and platforms prioritizes that 
iconography as essential to and as a visual equivalent of the media object 
being adapted (Newell 2017). Rather than grant the Handmaid a discern- 
able identity, the majority of book covers reinforce the anonymity of the 
Handmaid and, in so doing, establish in visual terms the condition of the 
uniform described by Atwood's narrator—to render the Handmaid both 
unseen and highly visible. As the image of the Handmaid transfers from 
edition to edition, the iconography of the costume gains semiotic weight. 
The visual iconic characteristics that emerge from the book covers initiate 
and reflect characteristics central to adaptations of The Handmaid's Tale in 
other media, including other print-based formats such as illustrated novels 
and graphic novels. 


2.5 PRINT-BASED MOVEMENT: ILLUSTRATION 
AND GRAPHIC NOVEL 


Book covers, book illustrations, and graphic novels share similar medial 
properties in that both are heavily semiotic and apprehended as two- 
dimensional, flat surfaces and they can share similar compositional and 
technical features and concerns, though they differ in degrees of iconicity 
and spatial-temporality. Book cover designs, as we have seen, are highly 
iconic and non-sequential—that is, they attempt to distill a book’s themes 
and tone into a single image, or may function as a “teaser,” a hint to spark 
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readers’ curiosity about a book’s contents. Book illustrations, on the other 
hand, typically adapt specific moments from a work, and invite readers to 
consider their semiotic import more directly in concert with the prose. We 
might think of the majority of book illustrations, particularly those pub- 
lished by major publishing houses and geared toward a mainstream mar- 
ket, as offering a limited sequentiality, in that most depict a succession of 
scenes from a novel that, when pieced together, convey an intentional 
trajectory or other system of cohesion. Graphic novels are fully sequential 
in that they communicate content visually through a series of intentionally 
linked visuals and frames. I now turn to consider the transfer of Handmaid 
iconography in Anna and Elena Balbusso’s illustrations for the 2012 Folio 
Society edition of The Handmaid’s Tale and Renée Nault’s 2019 graphic 
novel adaptation, The Handmaid’s Tale. 

The Balbussos utilize the established iconography of the red cloak and 
white bonnet, and explore, through composition and pictorial emphasis, 
the various significations of the uniform. The Balbussos transfer the color 
coding and primacy of the Handmaid from previous iterations through, 
they explain, the stylistic lenses of “Futurism, Russian Constructivism and 
fascist-period design"—styles that add a layer of emotional detachment 
and impersonalization to the Handmaid iconography and subtly under- 
score the novel’s social and political tensions (The Handmaid^s Tale 2019). 
Much like the 2010 Vintage cover design that presents a row of identical 
screen-printed Handmaid uniforms, the Balbusso cover focuses on the 
uniform: the Handmaid figure is depersonalized, rendered as a manne- 
quin. Four hands are positioned on the body of the mannequin as well as 
entering into the seemingly hollow lower-half of the torso, which symbol- 
izes the “empty vessel” to which Handmaids are expected to aspire. The 
putty-colored rectangle framing the figure is suggestive of a book cover— 
“the Handmaid’s Tale” within The Handmaid^s Tale—but its canting like- 
wise evokes the imbalance of the Handmaid within her environment. In 
keeping with the novel’s themes and those of other visual interpretations, 
this suite of six illustrations and a frontispiece explores tensions between 
Handmaids’ invisibility and heightened visibility. The Balbussos render 
figures within Gilead uniformly, in a manner that reinforces the deperson- 
alization of this stratified society. All Handmaids, Marthas, Wives, daugh- 
ters, Guardians, and so on look alike and are rarely distinguished visually. 
Yet the moments that the Balbussos select to adapt trend toward the per- 
sonal or interpersonal. In this way, they draw attention to the 
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oppressiveness of Gileadean frameworks and explore tensions between 
concealing and revealing characters’ “true” selves in opposition to 
that facade. 

Illustrations have the ability to direct vision, to render visible aspects of 
a narrative that might be downplayed by a writer or concealed intention- 
ally. Handmaids are fetishized throughout Atwood’s novel for all they rep- 
resent—fertility, sexuality, hegemonic power structures, and so on. The 
Folio Society edition includes an illustration of the narrator’s visit to the 
doctor’s office, titled “I take off my clothes, behind the screen, and leave 
them folded on the chair.” This image depicts the narrator behind a red 
accordion screen in the doctor’s office, unclothed save for her white bon- 
net and red stockings. This image is challenging in its positioning of the 
viewer and its messaging regarding the Offred. The novel makes clear that 
Handmaids remain clothed at all times, and that, even during the 
Ceremony, the body is still largely covered to help dissociate desire and 
other emotions from the act, to keep the Handmaid from being “pene- 
trated.” Seen in the context of the suite’s emphasis on uniformity, this 
image may provide a rare glimpse of individuality—the person distinct 
from the uniform. As the Balbusso Twins explain in an interview with 
Porter Anderson for Publishing Perspectives, “The Handmaids seem clones. 
Their faces are anonymous, they’re women emptied of their personalities, 
they've lost their identity [...]. When the Handmaid takes off her red uni- 
form, the body re-emerges with its carnality and identity" (Anderson 
2017). Yet the image also spectacularizes Offred through positioning and 
framing: the viewer is invited to see behind the privacy screen and to 
Observe Offred in a private moment of adjusting her stockings—a gesture 
fetishized in visual culture. 

As is the case with many illustrated novels, the Folio Society's illustra- 
tions are juxtaposed against prose different from their ostensible subject— 
à circumstance that invites and facilitates significations and thematic 
reflections otherwise unavailable. Within the Folio edition, “I take off my 
clothes" is juxtaposed not against the scene in the doctor's office, but 
against that in which the narrator is given a bath to prepare for the 
Ceremony: “I take off the rest of the clothes, the overdress, the white shift 
and petticoat, the red stockings, the loose cotton pantaloons." The narra- 
tor comments that, “My nakedness is strange to me already. My body 
seems outdated. Did I really wear bathing suits, at the beach? I did, with- 
out thought, among men, without caring that my legs, my arms, my 
thighs and back were on display, could be seen. Shameful, immodest” 
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(Atwood 2012: 62). These two moments of undressing—for the doctor 
and for the bath—evoke in Offred feelings of alienation and distanciation, 
of heightened visibility and objectification as they precede instances of 
physical violation—that of the doctor’s exam (and proposition of sex) and 
that of the Ceremony. The narrator states: “I avoid looking down at my 
body, not so much because it’s shameful or immodest but because I don’t 
want to see it. I don’t want to look at something that determines me so 
completely” (63). The Balbusso image, then, invites the reader to reflect 
on an aspect of Offred that she, herself, avoids. That the body visible in 
this image is so distinct from the uniformed body presented in other illus- 
trations may invite readers to also feel the strangeness of Offred’s naked- 
ness, and to reflect on which—uniform or individual—is the more 
determining or determined body. At a basic level, however, the image 
invites the reader to objectify Offred, to view her as a body. The viewer’s 
vision is thus aligned with that of Gilead’s dominant power structure, but 
differently—she is presented to us in posture that is coded openly as pri- 
vate and sexualized, whereas, within Gilead, she is coded as public and not 
sexualized. 

In 2019, The Handmaid’s Tale was adapted into a graphic novel by 
Renée Nault. Though the materiality of illustrated novels and graphic 
novels is similar, differences in compositional arrangement and in the pro- 
portion of word and images cause them to be perceived as distinct media. 
Commonly, the ratio of words to images in illustrated novels favors words, 
whereas graphic novels emphasize images, and tend to use the combina- 
tion of words and images to convey meaning. Nault utilizes the iconogra- 
phy of the Handmaid uniform to explore thematic tensions between 
Handmaids' visibility and invisibility, often to dramatic effect through her 
play with proportion, color saturation, and other graphic devices. The 
compositional conventions of the graphic novel allow the uniform to 
appear as “in motion"—moving through frames—and this movement 
increases the range of significations, as, for example, in the scene of the 
Particicution, Gilead’s communal execution event, which unfolds in a 
series of increasingly sharp and skewed panes in which the crowd of 
Handmaids blurs into a mob and the red of their cloaks becomes indistin- 
guishable from the red used to convey their rage and the blood of the 
victim. Nault uses the uniform symbolically, depicting it as blending into 
or standing out from the background to establish or reinforce themes or 
tensions. Nault’s interpretation draws from the color the stratification of 
Gilead society, and she describes Gilead itself as a “sort of a designer’s 


46 K. NEWELL 


dream because everything is colour-coded already and you just sort of 
have to guide the viewer with the symbols that already exist” (Dundas 
2019). In approaching Atwood’s novel, Nault explains to Sbondaland's 
Madeleine Deliee that *the comic format complements both the narra- 
tor's perspective—which reflects jumps in time, memory, and thought— 
and the story's frequent overlapping of reality and fantasy" (Deliee 2019). 
Nault hand-painted each page using watercolor, and selected distinct pal- 
ettes to convey setting and time: flashbacks to “the time before” are 
depicted through a full range of color that *evok[e] what we see every day, 
and a more normal time," whereas scenes in Gilead are depicted through 
a juxtaposition of muted backgrounds and high saturation, such as the 
vivid red, green, and blue of the Handmaids’, wives’, and Marthas’ uni- 
forms (Dundas 2019). 

Nault employs compositional arrangements familiar from previous 
visual iterations of The Handmaid’s Tale, as is evident in the striking two- 
page spread of the narrator and her shopping partner walking along the 
Wall, reminiscent of Marcellino's cover illustration. As is the case in 
Marcellino's design, the wall dominates both the composition and the 
Handmaids. The two women are indistinguishable, and identifiable only 
by their white bonnets, red uniforms, and shopping baskets. In Marcellino, 
the haunting quality of the composition is heightened by the Handmaid 
looking behind her at an unknown beyond the frame, whereas Nault's 
depicts both Handmaids moving forward toward an explicit horror: six 
figures hanging from the gallows—the images of fetuses on their neck 
placards identifying them as doctors. 

Given the nature of the graphic novel, one could say that the iconogra- 
phy of the Handmaid's uniform is reinforced in each panel that features a 
Handmaid. Nault, however, devotes several panels to Offred donning the 
Handmaid uniform at the novel's outset—slipping into the shoes, pulling 
on the gloves, adjusting the white wings, holding the shopping basket. As 
is the case in other adaptations, she extends this iconography to additional 
significations. Nault organizes the first page of *II: Shopping" into two 
horizontally oriented panels both of which depict the narrator's red shoes 
poised at the edge of the red circular rug of her bedroom in the 
Commander's house from two distinct perspectives. The top panel offers 
a bird’s eye view of much of the bedroom, which is rendered in grayscale 
save for the saturated red of the shoes and rug. The reader is able to see a 
window seat with a pillow embroidered with “faith,” the footboard of the 
bed, and a partial view of a door. The lower panel offers a close-up 
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perspective of the red shoes, which are centered in the composition. The 
text accompanying the lower panel is that in which the narrator considers 
her name, “I have another name [...]. I keep the knowledge of this name 
like something hidden, some treasure I'll come back to dig up, one day. I 
think of this name as buried? (Atwood and Nault 2019: II Shopping, 
n.p.). In Nault’s portrayal, the rug resembles a target, or, given the place- 
ment of the shoes at the edge, a hole to jump or fall into, or, further, given 
the juxtaposed text, a future excavation site. 

The Handmaid's shoes are a continued motif throughout the adapta- 
tion, and provide the focus of several panels—the narrator walking dou- 
bled with her shopping partner, under the stall door in the Red Center 
bathroom, dressing in her bedroom, walking through Serena's garden, 
shoes like hers dangling from the gallows. The rug also comes to symbol- 
ize the narrator and her personal safety. The concluding scene, in which 
the Eyes come for Offred, includes a panel in which a guard's boot steps 
onto the round red rug; yet, a few pages later, this potential threat is 
revealed as a potential opportunity. The novel's opening image is book- 
ended by that at the “tale’s” close—a full, white page, the top half of 
which depicts the narrator's billowing skirt and single shoe-shod foot as 
she runs toward the future, *And so I step up, into the darkness within; or 
else the light? (Atwood and Nault 2019: XV Night, n.p.). The absence of 
the rug, here, reinforces the idea that Offred has “stepped up," or 
“stepped into." 

As in the case of other adaptations, the graphic novel explores the ten- 
sion between the Handmaids' uniformity and individuality, both themati- 
cally and compositionally. Nault's achievement is perhaps best seen in a 
series of panels devoted to the narrator's daily meetings with her shopping 
partner, Ofglen. Atwood's first description of this meeting reinforces the 
manner by which Gilead codes Handmaids as identical and shows, also, 
that they remain individual to each other: *A shape, red with white wings 
around the face, a shape like mine, a nondescript woman in red carrying a 
basket, comes along the brick sidewalk towards me. She reaches me and 
we peer at each other's faces, looking down the white tunnels of cloth that 
enclose us. She is the right one" (Atwood 1985: 28-29). Nault adapts 
these meetings in several half-page panels spaced through the novel, each 
of which includes an inset half circle, which is divided into four rows, each 
depicting a distinct stage in the meeting: one Handmaid waiting alone, 
another's approach and their exchange of greeting, the two standing 
together, two walking off. The background behind the half circle depicts 
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the houses of two Commanders. The state of being the “right one” comes 
to mean later when the narrator realizes that she is meeting the “wrong” 
one. Toward the end of the novel, the narrator waits at the corner for the 
partner she has come to know as Ofglen. Nault renders this meeting com- 
positionally in the same manner as the previous two meetings. The accom- 
panying text reads: “I notice nothing at first. Then, as she comes nearer, I 
think that there must be something wrong with her. She looks wrong” 
(Atwood and Nault 2019: XIV Salvaging, n.p.). At this moment, the nar- 
rator realizes that the woman she has come to understand as Ofglen is no 
longer Ofglen. This name has come to signify a new body, one that 
Atwood’s narrator comes to understand through differential features: 
“She is altered in some indefinable way [...]. She isn’t Ofglen. She’s the 
same height, but thinner, and her face is beige, not pink” (Atwood 1985: 
294). Nault establishes Offred’s realization through visual markers that 
likewise become evident through comparison. Whereas in the previous 
scenes, the figure approaching from the left of the frame is slightly shorter 
than the figure waiting toward the center of the frame, in this last meeting 
the approaching figure is slightly taller than the figure waiting. In visually 
presenting this meeting as the same as those before it, Nault creates a tell- 
ing rift between the visual and worded cues and again reinforces that ten- 
sion between what is visible or invisible, what is observed, and what is 
understood. 


2.6 MorTIoNn-BASED TRANSFER 


As we have seen, the transfer of iconography in print-based visual adapta- 
tions of The Handmaid’s Tale tends to reinforce the Handmaid uniform as 
a shorthand signifier for a range of meanings, and does so in ways that 
foreground unique medial and modal aspects. I will now turn to an exami- 
nation of audiovisual, motion-based adaptations on stage and film. The 
transfer of iconography from a verbal, written media product to an audio- 
visual, motion-based one involves activities similar to those utilized in 
print-based visual adaptation in that worded descriptions and narrative 
voice are conceptualized in new ways, but audiovisual, motion-based 
adaptations likewise bring in additional considerations, among them ges- 
ture, movement, three-dimensionality versus two-dimensionality, live or 
recorded performance, live or recorded speech and music. For example, 
The Handmaid’s Tale as conveyed through ballet poses a medial experi- 
ence unique from an operatic version, though both communicate broadly 
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through set, sound, movement, and gesture. Similarly, a one-woman the- 
atrical performance of The Handmaid’s Tale and a television series both 
transfer theme, iconography, and other elements from Atwood’s novel 
and other adaptations, yet each poses unique challenges, among them pac- 
ing, character empathy, dialogue construction, and set design. My goal in 
what follows is not to provide an exhaustive examination of each modality, 
but, rather, to highlight the manner by which balletic, operatic, theatrical, 
filmic, and televisual adaptations of The Handmaid’s Tale interpret ico- 
nography of the Handmaid uniform, and, in so doing, demonstrate the 
flexibility of the iconography and the manner by which it might be bent to 
align with a specific modality. 

As has been the case for print-based adaptations, the transfer of iconog- 
raphy from one media product to another in audiovisual, motion-based 
adaptations has come to rest most visibly in the uniform. As the uniform 
transfers from motion-based adaptation to motion-based adaptation, it is 
adapted to meet the requirements of different modalities. Because the 
uniform described by Atwood is too restrictive for dance, in adapting The 
Handmaid’s Tale to ballet, costumer designer Liz Vandal reimagined the 
Handmaid uniform for choreographer Lila York's *dance drama," which 
premiered with Canada's Royal Winnipeg Ballet in October 2013, as a red 
tunic with a triangle-shaped yoke that covers the head, and is open at the 
sides and fitted over a gray leotard. The bonnet and white wings are rei- 
magined as a structured, pyramid-shaped headpiece fitted to the back of 
the dancer’s head. While a technical necessity for this modality, the 
increased mobility and unrestricted vision of this uniform create a version 
of the Handmaid distinct from that offered by Atwood, and many other 
adaptations. Within Gilead's stratified society, mobility is linked to power, 
and each group's uniform allows a degree of mobility and visibility appro- 
priate to their station. As Atwood's narrator explains, the Handmaid's 
uniform limits perception and vision: “Given our wings, our blinkers, it's 
hard to look up, hard to get the full view, of the sky, of anything. But we 
can do it, a little at a time, a quick move of the head, up and down, to the 
side and back. We have learned to see the world in gasps" (1985: 40). In 
tokenizing a fundamental feature of the costume, the ballet impacts the 
audience's sense of how the iconography functions within the society. 

The conditions of live performance may have informed also the deci- 
sion of the Boston Lyric Opera adaptation (2019), directed by Anne 
Bogart, and conducted by David Angus, to opt for the white bonnet, but 
not the wings. The wings would likely impact the performers’ hearing 
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and ability to communicate with each other on stage. Yet, as with the bal- 
letic adaptation, the visibility of the Handmaids' faces to audiences both 
within and outside the diegesis, and Handmaids’ own unobstructed vision, 
impacts audiences’ understanding of the role of the Handmaid uniform 
within Gileadean social structures. In other ways, though, the opera 
overtly aligns itself with established iconography of the uniform by includ- 
ing in the program a sketch for the Handmaids’ costume by James 
Schuette, who also designed the sets, juxtaposed against an image of 
Marcellino's cover design for the Jonathan Cape edition.” 

The one-woman The Handmaid's Tale stage adaptation written by 
playwright Joe Stollenwerk's script, directed by Brian Isaac Phillips, and 
performed in 2015 at the Know Theatre of Cincinnati, featuring Corinne 
Mohlenhoff as Offred, likewise omits the white wings, though, as the play 
unfolds within a single, private setting, the absence is not as pronounced. 
Additionally, given that the performance (including communication of 
character, story, narrative tension, and conflict) rests on the nuances of 
Mohlenhoff’s delivery, the costume decision is understandable. In Noelle 
Wedig’s costume design, the bonnet fits over a cowl-like garment that 
conceals the Handmaid’s hair and head. This layering of garments coupled 
with Andrew J. Hungerford’s strategic lighting design, which utilizes 
high- and low-contrast effects to reflect and craft emotional and environ- 
mental mood, creates a Handmaid who, though literally visible to the 
audience, comes through as metaphorically veiled. Zack Hatfield, review- 
ing the play for The News Record, likewise comments on the relationship 
between costuming and lighting in this production, calling Wedig’s inter- 
pretation of the Handmaid costume, “a visual marvel, encapsulating the 
appropriate shades of paranoia and violence,” and commenting that 
Hungerford’s lighting “creates a tonal dissonance vital to understanding 
the complex psyche of Mohlenhoff’s role” (Hatfield 2015). 

Implications for performance and reception likewise inform how film 
and television adaptations interpret the Handmaid uniform. Colleen 
Atwood's take on the iconic uniform for the 1990 film adaptation directed 
by Volker Schlóndorff and starring Natasha Richardson is much more 
streamlined than other iterations. Handmaids in this version wear only a 
sheer, red head scarf in public, and occasionally a red veil. Handmaids’ 
faces are visible, and they remain distinguishable from one another, as a 
result. The decision to avoid a more obtrusive head covering may have 
come from a fear of blocking the audience's view of actors’ faces. If the 
poster advertising the film, which features lead Natasha Richardson, 
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ostensibly unclothed save for the red garment she holds against her body, 
is any indication, this film seems much less concerned with highlighting 
the Handmaids’ invisibility within Gilead than in highlighting Richardson’s 
star qualities. By contrast, for the Hulu series created by Bruce Miller, 
which aired on April 26, 2017, costume designer Ane Crabtree, under- 
standing the aesthetic and symbolic impact of the full, white wings, per- 
suaded the show’s developer to use them instead of less obtrusive 
headscarves (Chuba 2018). In the third season, in particular, the series 
begins to exploit the headpiece as a device of both invisibility and visibility. 
Episode 2, for example, opens with a scene of June/Ofjoseph (formerly 
Offred) waiting for her shopping partner, and watching other Handmaids 
pass her on the street. The camera adopts a subjective perspective from 
“behind” the white wings, allowing viewers to see the scene as she does, 
as if through a tunnel. Once within the grocery, however, June/Ofjoseph 
uses the privacy provided by the wings to exchange information with 
Alma/Ofrobert, a Handmaid involved in helping to organize the 
resistance. 

The motion-based adaptations reiterate the significance of the iconog- 
raphy of the red cloak and white bonnet in ways that highlight features of 
their specific modalities. Thus, the highly stylized uniform adopted by the 
ballet supports both the aesthetic characteristics and physical necessities of 
the medium—the uniform is visible, but downplayed to foreground the 
physicality of dance. In comparison, the Hulu series can utilize more 
restrictive uniforms because the handheld cameras can compensate for 
actors’ restricted mobility. 


2.7 TRANSFER AND NETWORKS 


The popularity of the Hulu series and the popular media’s depictions of 
the series as a mirror to the political climate have contributed, no doubt, 
to the “real world” upswing in Handmaid iconography and what we might 
think of as “embodied” adaptation. Since spring 2017, individuals have 
donned Handmaid cloaks and white bonnets at numerous protests glob- 
ally, mostly related to women’s reproductive rights and real and threat- 
ened erosions of other freedoms, among them the NARAL Pro-Choice 
protests in Texas, protests against the abortion ban in Ireland, and against 
Brett Kavanaugh's 2018 United States Supreme Court nomination. As 
Guardian writer Arielle Bernstein notes, “The image of the red robed and 
white bonneted women [...] has become as celebrated as Rosie the 
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Riveter” (2018). Alex Yarde, writing for The Good Men Project, compares 
“the unmistakable iconography of the red bonnet as a universal symbol of 
oppression,” to “the white worn in the Suffrage Movement at the turn of 
the last century” (2019). This pattern of reading the Handmaid “in con- 
text” brings us to an important question: what does the Handmaid uni- 
form symbolize? 

To answer this question, I return to the examples with which this essay 
began and the questions they raise: How did a line of red trees come to 
make a political statement? How did a group of women in red cloaks come 
to illustrate a threat to scientific research? How did a red cloak and white 
bonnet become a site of sexual pleasure? Each of these examples comes to 
“mean” at the point of transfer, or, borrowing from Ellestróm's commu- 
nication model, in the "intermediate stage" between one place and 
another, *that makes the transfer possible" (2020: 10). That is, each 
comes to mean at the point when the perceiver recognizes the “thing” as 
not the thing. For example, the trees in the colonnade are simply topiaries, 
three-dimensional static structures composed of faux cranberry branches. 
Set within the larger cultural moment, the trees are still seen as trees but 
also, satirically, as not trees, as Handmaids. In transferring the iconogra- 
phy of the Handmaid from the Hulu series to the holiday display, the 
edited photograph discloses a semiotic message that conceals the material- 
ity of the topiary. The materiality continues to exist (i.e., the topiary con- 
tinues to exist as a holiday decoration), but, within the context of the 
photograph, the tree's semiotic potential dominates in the transfer of the 
message to the perceiver. The political efficacy of the protesters dressed as 
Handmaids comes in the recognition that they are not Handmaids, but, 
rather, are evoking the status of Handmaid to prove a point. Were the 
women actually Handmaids, the act of wearing the uniform would not 
constitute protest. Likewise, in the example of the “Brave Red Maiden” 
costume, the iconography of the Handmaid transfers to the person wear- 
ing the costume. A pleasure in the costume is recognizing the lover as 
both Handmaid and not Handmaid. In each example, the transfer of ico- 
nography exposes the border between the thing and not the thing. The 
precision of the semiotic sign is not as important as the sign’s broader 
connotations, which are understood through the perceiver's recourse to 
collateral experience. 

Perceivers’ collateral experience, which, in the case of these examples, 
takes shape from a larger adaptation network, might include familiarity 
with Handmaid iconography through experiences with one or more of the 
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numerous print- and motion-based adaptations, visual publicity for the 
Hulu series, or accounts of protesters in the news. The experience allows 
perceivers to perceive the message in the transfer, thus enabling them to 
decode the representation—as a political or erotic, for example—within 
the context of The Handmaid’s Tale. Perceivers’ experiences with varied 
messaging allow them to decode the line of red trees, or the group of 
women in red cloaks, or the costume denotatively, and perceivers’ recourse 
to the intermedial adaptation network allows them similarly to decode 
these examples connotatively. 

What can The Handmaid’s Tales adaptation network tell us about 
intermediality in general? As Rajewsky, Ellestróm, and others have demon- 
strated, the so-called borders between media and modalities are largely 
contrived and perceptual. Recognizing such borders as constructed, how- 
ever, is not in conflict with a recognition of their usefulness or a curiosity 
related to their areas of porosity—the shifting points of contact generated 
through processes of adaptation. Perceivers’ collateral experiences shape 
their decoding of icons, as does the generalized collateral experience gen- 
erated by the repetition of specific iconography through an adaptation 
network. Ellestróm's and Rajewsky's proposals that intermediality be con- 
ceptualized as a meeting of “border zones” remain a productive way of 
thinking about such relationships (Ellestróm 2010; Rajewsky 2010). As 
Rajewsky explains, borders can be understood as sites of possibility, “as 
enabling structures, as spaces in which we can test and experiment with a 
plethora of different strategies" (2010: 65). Adaptation is the enabling 
structure that makes the transfer of iconography between media products 
possible. Much like Atwood's narrator, who serves as an intermediary 
between books and discs, but who also redirects that transfer to other, 
unanticipated points of communication, adaptation illuminates areas of 
porosity between media and media products, further erodes notions of 
impermeable divisions between media, and enables understanding of 
broadening intermedial networks. 


NOTES 


1. Yandy pulled this costume, also dubbed the “sexy Handmaid,” from the 
website after negative public response. See Elassar and Muaddi (2019) and 
Vagianos (2019). 

2. David Lyman, for example, commenting on Strollenwerk’s 2015 play states 
that “There was a time [...] when the idea of suspending or abridging 
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Constitutional rights was inconceivable. Yet in the wake of the 9/11 attacks, 
we saw just how natural it felt for citizens and governments to overreact in 
the name of national security” (Lyman 2015). Deborah Dundas, who inter- 
viewed Renée Nault for The Toronto Star, notes that delays in developing the 
graphic novel adaptation resulted in a keen alignment of the book's themat- 
ics and current politics: *its dystopian slant echoes many of the issues facing 
a Donald Trump America—eroding women's rights, the rise of religious 
fundamentalism? (Dundas 2019). 

3. Rajewsky quotes from Werner Wolf (1999: 40-41). 

4. I recognize that metaphors of movement are contested in adaptation stud- 
ies, as they elide relationships between form and content. I adopt the meta- 
phor of transfer deliberately here in recognition of its significance in The 
Handmaid's Tale. “Transfer” is also not without precedent in discussions of 
adaptation and intermediality. Lars Ellestróm uses this metaphor effectively 
in presenting what he identifies as “ten border zones of adaptation” 
(2017: 513). 

5. When asked about her inspiration for the Handmaids’ uniforms, Margaret 
Atwood cites the stick-brandishing woman on containers of Old Dutch 
Cleanser—a figure who “chases dirt” and promises “healthful cleanliness,” 
and who terrified Atwood as a child. Atwood’s adaptation of this image rests 
largely on the headpiece that hides the figure’s face from view (Flock 2017). 

6. The Handmaid’s Tale opera was first performed in 2000 by the Danish 
Royal Opera in Copenhagen, conducted by Michael Schonwandt. In 2003, 
the English National Opera and the Minnesota Opera both reprised the 
program. In 2004, the Danish Royal Opera performed the program in 
Toronto. 

7. That this production is staged in the Ray Lavietes Pavilion—the Harvard- 
owned athletic arena thought to be the inspiration for the building that 
houses Atwood’s Red Center—with the action unfolding on what would be 
the gymnasium floor, points to an attempt to foster an immersive experi- 
ence. As Lucy Caplan notes in an essay included in the production program: 
“rather than just reading about the hypothetical possibility of this space as 
the Red Center, you experience that transmogrification in real time and 
among other people. It is happening here” (Caplan 2019). 
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3.1 ASSEMBLING FOUNDATIONS 


It is possible to explore the relationship between architecture and other 
media through a wide variety of theoretical scopes—from time-space rela- 
tions, as described by Gaston Bachelard (1994 [1964]) and Stephanie 
Glaser (2009, 2014), to Ulf Pettersson’s cognitive approach (2013), or 
even Jennifer Bloomer's philosophical one (1993). It is also possible to 
view the architect's activities or a piece of architecture from different 
angles. Although still not nearly as extensive as the field of research dealing 
with the relationship between literature and painting, investigations of 
literature and architecture have nonetheless become more common over 
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recent decades. Departing from the premise that ekphrasis is meant to 
evoke an absent image in the mind of the spectator, or the reader, and 
thereby to provoke an emotional response, I have elsewhere studied such 
relations by using the notion of architectural ekphrasis as a theoretical 
operator, under the light of intermedial studies (Vieira 2016, 2020). While 
examining the possibilities for working with this typology, I realized that 
the already consolidated pictorial models’ parameters were not sufficient 
for my needs. One of the greatest challenges I faced was the delimitation 
of architecture’s medial traits that literature is actually able to transmedi- 
ate. Lars Ellestróm's proposed model for studying the transfer of media 
characteristics was crucial to achieving consistent results (2010, 2020). 
The four modalities of media, along with their qualifying aspects, became 
the backbone of an interpretative model proposed to explore the presence 
of architecture in literature. The aim of this chapter is then to delve into 
the core of my research: the modality modes of architecture. 

According to Mario Biselli, architecture is “the intersection between 
the hard sciences and humanism”! (Biselli apud Penna 2005: np), while 
Gustavo Penna defines it as “the spacial synthesis of all culture"? (Penna 
2005: np). In order to establish which kind of architecture I mean to 
explore, I depart from its dictionary definition as the “art and technique 
of organizing spaces and creating environments to shelter the many kinds 
of human activities, aiming also at specific aesthetic intentions"? (Houaiss 
and Villar 2009: 186), along with the premise that, in architecture, “the 
work of many individuals will be involved and transformed within the 
unity of the finished whole” (Crowther 2009: 182-183), where the 
“whole” is the building itself. Adopting a more philosophical point of 
view, Karsten Harries reflects upon the notion of architecture as an inter- 
pretive and critical act, as well as on the possibilities of viewing architec- 
tural works as “texts” (Harries 1997: 4). According to Harries, the style of 
a building, when attributed to a certain code, can communicate a “way of 
standing in the world, a specific ethos” (Harries 1997: 92). The French 
architect Emile Aillaud (1902-1988) has for his part argued that “archi- 
tecture is nothing more than the organization of a story whose syntactical 
elements are constituted by a scenery that carries it”. Aillaud explains that 
he himself has built “cities, and these cities are themselves operas, stories 
told; they are great stories in which one can dwell, an inhabitable story” 
(Aillaud apud Hamon 1992: 29, footnote 13). Based on Aillaud's reason- 
ing, Philippe Hamon illustrates the important narrative capability of archi- 
tects, ^who on the one hand listen to the narrative instructions of their 
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clients, and on the other hand produce places that are stories as well as 
stories that are places” (Hamon 1992: 29). As a qualified medium, archi- 
tecture may, under specific contextual qualitative aspects, convey messages 
to society, such as power or status. With this idea in mind, I consider that 
the two-way narrative capability suggested by Hamon requires a refine- 
ment of the communicative tools used in interactions between the client, 
the architect, and other relevant professionals involved in the architectural 
process, in addition to the building's final users. 

Considering how all media products necessarily have several media 
dimensions, Jorgen Bruhn uses the term medialities to discuss media as 
“specified clusters of communicative forms” (Bruhn 2016: 17) that “may 
be briefly defined as tools of communicative action inside or outside the 
arts” (1). Accordingly, in his treaty on the autopoiesis of architecture, 
Patrik Schumacher* argues that architecture can and should be considered 
as a (qualified) medium (2010: 323-362). The author explains that 
“architects communicate to wider audiences via buildings and designed 
spaces. Buildings and spaces constitute a very specific type of communica- 
tion: they are ordering and framing communications [...] between people 
co-present in space” (Schumacher 2011: 1). In other words, when under- 
stood as a process, architecture may be considered a medium due to its 
inherent potential to store communication. The concept of “process” is 
thus central to my previous investigation of architectural ekphrasis.? 
Moreover, the communication between those involved (client, architect, 
engineer, technicians, construction workers, and others) in relation to the 
edifice’s modality modes is key to the fulfillment of an architectural 
process. 

Architecture, as a process, originates in a client's wish. And that wish 
must, before it can be consolidated into a palpable building, undergo a 
succession of stages carried out through sketches, technical drawings, 
mockups, models, and so forth. Such depictions operate in the same way 
as /ibrettos in opera, musical scores in performance, or screenplays in film. 
As such, we understand that architecture is developed through a dynamic 
communicative process in which every stage is accomplished through spe- 
cific qualified submedia (graphic and technical representations) that not 
only “crave” for transmediation but also operate as part of a predeter- 
mined hierarchical process. For instance, the client’s desire is followed by 
the architect’s interpretation of the oral verbalization of such wishes, 
expressed through sketches and technical drawings. However, as the archi- 
tect Steen Eiler Rasmussen states, “not everyone [...] can visualize a 
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building merely by looking at the plans” (Rasmussen 1964 [1959]: 9). 
Therefore, the inclusion of other submedia—such as models and technical 
representations—is made necessary to the process’s successful culmination 
at the construction site, where the architectural work will in fact be 
executed. 

According to Rasmussen, “the architect is a sort of theatrical producer, 
the man who plans the setting for our lives, [...] but his producer job is 
difficult for several reasons. First of all, the actors are quite ordinary peo- 
ple” (Rasmussen 1964: 10). In this sense, the submedia which structure 
the stages of the process “are not an end in themselves, a work of art, but 
simply a set of instructions, an aid to the craftsmen who construct his 
buildings” (Rasmussen 1964: 14). These stages can be roughly described 
as: (a) the verbalization of a client’s desire; (b) the architect’s mental inter- 
pretation of the client’s intention; (c) the representation of this cognitive 
import through two-dimensional visual qualified submedia; (d) the 
graphic representation of the future edifice’s volume, expressed by CG 
renderings or 3D models;? (e) the qualified submedia known as a blue- 
print, comprised of plans, facades, and sections, technically and diagram- 
matically representing the architectural project;? (f) the construction site, 
in which the entire process will be transmediated, to finally culminate; (g) 
in the building per se. All of the mentioned submedia— graphic, diagram- 
matic, and technical representations, such as sketches, plans, blueprints, 
mockups, CG renderings, and 3D models—are relevant within the chain 
of a communicative process containing different levels of technical and 
architectural literacy, where a considerable amount of verbal articulation 
may be required if any of the stages is not decoded in detail. 

Once the understanding of architecture as a medium has been estab- 
lished, I believe it is useful to identify its intrinsic characteristics and affor- 
dances, which are the medial traits that may be transmediated by literature, 
more specifically, works of fiction in which architecture plays a significant 
role so that the full architectural experience can be unfolded in the read- 
er's mind through words. Since I have previously identified that the already 
consolidated pictorial models? parameters are not enough for scrutinizing 
the relationship between these two media types—architecture and litera- 
ture—the proposed model for the study of transfer of media characteristics 
among dissimilar media in this volume (Ellestróm 2020) was crucial for 
the delimitation of architecture's medial traits. Therefore, as this is a 
reductionist model, where there is nothing to remove, but certainly to 


3 BUILDING BRIDGES: THE MODES OF ARCHITECTURE 63 


expand from, the foundation of my analysis also counts on the notions of 
embodiment and perspective in order to broaden the understanding of the 
process in which architecture is transmediated by literature. 


3.2 SETTING UP THE PIERS, OR THE MODES 
OF ARCHITECTURE 


As proposed by Ellestróm (2010, 2014, 2018a, b, 2020), semiotic charac- 
teristics—which include iconic, indexical, and symbolic aspects—are com- 
bined with presemiotic ones—which speak to the fundaments of mediation 
by involving the material, sensorial, and spatiotemporal modalities and 
their respective contextual and operational qualifying aspects. Therefore, 
in order to investigate the four modalities of architecture, I depart from 
questioning the basics: (a) What is the dominant sign system in architec- 
ture? (b) What materials constitute architecture? (c) How can architecture 
be perceived by the senses? (d) What are the possible relationships between 
architecture, time, and space? (e) Under which historical, cultural, and 
social circumstances has the building been designed? (f) Which aesthetic 
movement has influenced the architectural process? 

The semiotic modality deals with the system of media signification and 
representation; the representing and represented; and more specifically to 
my case study, the objects and different media products that act as signs in 
relation to one another. Thus, it is relevant to understand how iconicity is 
based on the similarities between the involved media types; indexicality on 
their contiguity; and symbolicity on pre-established habits and conven- 
tions (Ellestróm 2010, 2014, 2018a, b, 2020). The first premise, the 
structural similarity—or diagrammatic iconicity, according to Charles 
Sanders Pierce—between the style of the architectural work and the writ- 
ten text, is probably the most frequently studied, as for instance in the way 
that nineteenth-century English Gothic fiction borrowed elements from 
gothic medieval architecture. Hamon (1999) consistently shows examples 
of iconicity based on similarity in his work on the relations between archi- 
tecture and nineteenth-century French literature.'° Regarding the archi- 
tectural process stages that take place before the construction begins, the 
second premise relies on indexicality, whereas the habits and conventions 
suggested by the last premise will depend on the cultural background of 
those involved. Hence, architecture as a process is apt to prompt a com- 
municative process by encompassing both visual and verbal signs, by 
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means of sketches, plans, and models, as well as through the negotiations 
between client, architect, and all other professionals involved in the design 
project. Therefore, meaning is assigned through a combination of signs of 
equal importance, weight, and value within the architectural chain of 
semiosis. 

An edifice is comprised of different materials, such as brick, cement, 
mortar, glass, and steel. However, the material modality concerns both the 
physical materiality and the mental relationship involved within its trans- 
mediation. Therefore, if architecture is to be understood as a process span- 
ning several stages, materiality will also manifest itself inside the minds of 
those involved in said process, as discussed by cognitive studies. Architect 
Christian Norberg-Schulz (1980) relates the materiality of architecture to 
the linguistic structure employed in its procedural verbal negotiations. 
According to him, the structure of a place—be it a country, a landscape, 
or a building—includes the space itself, the characteristics of the man- 
made environment, and the respective technology employed in its execu- 
tion. The author states that places tend to be designated by nouns, such as 
“island”, “forest”, or “street”, as well as “wall”, “roof”, and “door”, since 
instead of referring to “space” in our daily lives, we refer to the relation- 
ships between things which are, for example, “over”, “in”, or “along” one 
another. Space, therefore, if understood as a system of relationships, is 
indicated through prepositions and phrasal prepositions. The technology 
and the characteristics present in man-made environments are, in turn, 
indicated by adjectives (Norberg-Schulz 1980: 15-16). Norberg-Schulz's 
contribution is indeed useful in the linguistic analysis of architectural 
ekphrasis. 

As a process, architecture comprises a series of stages between the deci- 
sion of having something built to the actual usage of the edifice in which 
is created an interactive discourse between client(s), architect(s), and all 
others involved. Beyond the linguistic elements put forth by Norberg- 
Schulz, this communication will only reach a common goal when a com- 
mon cognitive import is incited among all parties. Although the process is 
conceived in the client's mind, its function will be triggered by the archi- 
tect. This means that the transfer of vivid visions and perceptions by means 
of a shared mental image, which happens to be the rhetorical principle of 
ekphrasis, is highly relevant to its successful accomplishment. 

According to Rudolf Arnheim, “a building, [...] in all its aspects is a fact 
of the human mind. It is an experience of the senses of sight and sound, of 
touch and heat and cold and muscular behavior, as well as the resultant 
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thoughts and strivings” (Arnheim 1977: 4). Thus, when it comes to the 
sensorial modality, it is imperative to investigate the effects of the five 
senses, along with proprioception (and even interoception), on the pro- 
duction and reception of an architectural site. The sensorial modality 
relates to the reception of the final product. After all, many actions must 
be taken in relation to an edifice’s affordances,!! such as the opening /clos- 
ing of doors to enter/exit spaces, or the opening of windows to let light 
and air in/out. Moreover, regarding touch, the impact of the relative 
brightness, temperature, and humidity in an edifice will all affect its users. 
The combination of multiple materials with the natural element—meteo- 
rological conditions—allied to the human activities happening inside the 
edifice, will produce aromas and odors which are likely to affect the sense 
of smell. One example might be the smell of food that recurrently comes 
from the kitchen and is permanently absorbed by the living-room carpet. 
Hearing is also affected by the combination of the chosen materials, the 
weather, and human activities: for example when the sound of rain on a tin 
roof hinders someone’s concentration while studying. The only sense that 
is not likely to manifest in an architectural process is taste. However, 
regarding its connotative meaning, taste, as in aesthetics, should also be 
considered. For Paul Crowther, “the building’s articulation of form, shape, 
and mass in enduring materials, on the basis of symmetry, proportion, and 
the like, offers a kind of rectification or idealization of the body’s vectors 
of sensorimotor activity” (Crowther 2009: 181—author’s emphasis). 
Although not as deeply scrutinized as the five senses, along with mental 
introspection and interoception, the relevance of proprioception is brought 
about in the expanded model for understanding intermedial relations 
(Ellestróm 2020: 27, 30, 49). The kinesthetic sense, in particular the sen- 
sorimotor activities, must be included when it comes to the investigation 
of architecture's sensorial modes. As underscored by Arnheim, all these 
sensorial experiences depend on the historical, social, and even personal 
context. In sum, an architectural site can virtually activate all senses at once. 

Given that “spatiotemporal modality is a category of spatiotemporal 
media modes" and media products “consist of physical matter", their spa- 
tiotemporal properties can “be grasped by human minds” (Ellestróm 
2020: 48). An architectural process definitively occupies different types of 
spaces which cannot be dissociated from time. In spite of being known as 
the art of creating space, it is perhaps more accurate to say that architec- 
ture alters and organizes space. Unsurprisingly, space and time are among 
the comparative aspects most frequently studied by both architectural and 
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literary scholars. Rasmussen argues that the relationship between space 
and time is among the greatest difficulties faced by any architect whose 
“work is intended to live on into a distant future”. The professional “sets 
the stage for a long, slowmoving performance which must be adaptable 
enough to accommodate unforeseen improvisations” in an edifice which 
“should preferably be ahead of its time when planned so that it will be in 
keeping with the times as long as it stands” (Rasmussen 1964: 12). For 
him, “one of the proofs of good architecture” lies in buildings whose 
space “is being utilized as the architect had planned” throughout time 
(Rasmussen 1964: 12). 

In architecture, communication and human experience do not endure 
virtually, as in literature; they happen, effectively, within a materialized 
space. In contrast with literature and cinema, which can be preserved 
through technical, digital, or virtual reproduction, an edifice never reaches 
the status of final product, since its materiality will always be vulnerable to 
climatic and socioeconomic variations. Another noteworthy aspect of the 
spatiotemporal modality is the ratio between the final product's volume, 
depth, and height, as considered by Stephanie Glaser in her latest works 
(2014, in press). 

In order to provide stability to the piers, besides the understanding of 
architecture's four modalities, special attention should be given to the ori- 
gins, delimitations, and usage within the specific historical, cultural, and 
social circumstances of both production and reception of the investigated 
architectural environments in relation to their aesthetic and communica- 
tive features, as in the qualifying and operational aspects suggested by 
Ellestróm's model (2020). This investigation leads to the fact that, due to 
the matter of form and function, including the study of ergonomics and 
affordances of architectural elements, the scale of an edifice is set by the 
human body. Thus, two more notions inherent to architecture should be 
taken into consideration in order to tangibly delve into its modes: embodi- 
ment and perspective. 


3.3 ADDING GIRDERS: EMBODIMENT AND PERSPECTIVE 


In section two, I discussed how the modes of architecture both operate 
throughout its process and compose its final media product. Since its 
materiality is also manifested at the virtual level, the semiotic mode may be 
indicated not only by possible structural similarities and diagrammatic 
forms of representation evidence, but mainly by verbal negotiation in 
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assigning meaning along the architectural process as a whole. By delving 
into the presemiotic modes, I realized the indissoluble three-dimensional 
space and time in relation to the building, and the role of affordance and 
ergonomy within aesthetic matters of the studied edifice. After all, since 
ancient times, architecture has relied on the human body as both meta- 
phor and unit of measurement (Spurr 2012: 36). Yet the interaction of the 
user with architectural environments through the senses, including pro- 
prioception, makes the notion of embodiment indispensable to set a back- 
bone for the architectural traits most easily transferred to literature. At the 
same time, the points of intersection of the different connotations of the 
notion of perspective in different disciplines—architecture, art history, and 
literature—must also be taken into account. 

The understanding of how subject and object, in opposition to 
Cartesian dualism, comprise two facets of a single unit is fundamental to 
further investigations of architecture as a process. The concept of embodi- 
ment proposed by Maurice Merleau-Ponty (1945) sustains not only the 
inextricability of body and mind, but also the awareness of the perception 
and representation of the world (Auslander 2008: 136-139). Embodiment 
is used as a basis for performance studies when dealing with stage experi- 
ence by the actors and its perception by the spectators. As previously men- 
tioned, architects Emile Aillaud and Steen Rasmussen established, in two 
distinct historical moments, the analogy of the building as stage. Chiel 
Kattenbelt (2006), in a similar line of reasoning, uses the term “corporeal- 
ity” to designate the effect of the actor’s body onstage as in the materiality 
of that body in the theatrical space. This idea recalls the proposed discus- 
sion of how the body of a dancer may act as both technical medium and 
media product, as when Ellestróm quotes the famous question posed by 
Yeats—“How can we know the dancer from the dance?”—in the introduc- 
tory chapter of this volume (2020: 36). Nonetheless, in the case of archi- 
tecture, the focus should not be on distinguishing the architect from the 
edifice, but on determining how anyone can actually experience kines- 
thetically the three-dimensionality of its final media product. That is to 
say, even if this process initially relies on concepts and transfer of mental 
cognitive import between client, architect, and other professionals, in 
order for any edifice to be considered architecture, as opposed to a piece 
of sculpture, it must eventually be experienced by the bodies of final users. 

Inan attempt to advance relevant aesthetic and ethical discourses within 
architectural theory, Asa Dahlin (2002) highlights that it is not enough to 
(re)cognize architecture's material aspects. For her, reason and emotion, 
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much like mind and body, should not be treated as oppositions, but as 
parts of the same system. It is therefore necessary to consider knowledge 
generated by humans’ physical as well as mental consciousness. Sophia 
Psarra (2009), in turn, is concerned with the way theoretical focus on the 
visual perception of form has shifted to the embodied experimentation of 
space (13), which guided her methods for investigating the interfaces 
between conceptual and perceptual facets of an architectural project to be 
based on the notion of embodied experience. In sum, embodiment—as in 
proprioception, be it from a spatiotemporal and sensorial understanding, 
or from a philosophical standpoint—is relevant when showing the effect of 
the user's body within an architectural environment. 

While Rasmussen states that *it is not enough to see architecture; you 
must experience it [...] you must dwell in the rooms, feel how they close 
about you, observe how you are naturally led from one to the other? 
(Rasmussen 1964: 33), Arnheim calls attention to the fact that optical 
images are projected in two dimensions by the human eye, so the brain 
does fully grasp a three-dimensional object just from one stand point 
(Arnheim 1977: 110). To provide a more concrete example, the way in 
which “we may change our position deliberately to gain a more compre- 
hensive view" (110) can be illustrated by the renowned passage from 
Marcel Proust’s In Search of Lost Time in which the narrator observes 
"three clock towers that change their relative position" (118-119). 
According to Arnheim, in this scene, Proust anticipates the cinematic 
technique of traveling, which is related to periegesis, an ancient rhetorical 
tool employed in the description of places, landscapes, and buildings (see 
Glaser in press). Accordingly, Rasmussen suggests that users? perception 
of different standpoints is enabled by walking around and within an archi- 
tectural site. So even if perspective deforms the object or tricks the eye, the 
human mind still *organizes, completes and synthesizes the structure? 
such that the three-dimensional object *will be seen spontaneously as a 
whole? (Arnheim 1977: 110). The acknowledgment of the timeless exis- 
tence of an edifice in space in the visitor's architectural experience, along 
with the peculiarities of the human eye in relation to the mind, heightened 
my awareness of the relevance of the dialogue between the notions of 
embodiment and perspective. As previewed, I now present how the differ- 
ent facets of the notion of perspective affect architecture's medial traits to 
be presented by literature. 

In architecture studies, even if it is part of a wider discussion of repre- 
sentation at a more technical level,’ Alberto Pérez-Gómez and Louise 
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Pelletier have claimed that their “ultimate aim is to probe the possibilities 
of building architecture as a poetic translation, not a prosaic transcription, 
of its representation” (Pérez-Gómez 1997: 8). The authors have histori- 
cized methods of architectural representation since the Renaissance, when 
the introduction of systematized descriptive geometry significantly shifted 
the terms between the architectural process and its final product (Pérez- 
Gómez and Pelletier 1992: 22-37). Another major shift in the role of 
perspective within the architectural process was promoted by the con- 
sciousness of the observer as the organizer ofa composition (Pérez-Gómez 
1997: 19). In the same vein, Karsten Harries calls attention to the way 
architecture “relies on imitation” (1997: 112) as do other art forms. These 
theorists’ discussion of the evolution of forms of representation has con- 
tributed to my own understanding of the difference between architectural 
technical representation by means of perspective, as well as the notion of 
mimetic representation prevalent in traditional ekphrastic studies. 

The origin of the term perspective in art history and the history of sci- 
ence derives from either Arabic geometry (i.e. math) or western percep- 
tion (i.e. aesthetic theory) (Belting 2011: 1-2). Math plays an important 
part in painting and architecture, which is not necessarily true for literary 
studies. Despite being “unoccupiable” (10), the vanishing point of classic 
perspective in painting allows spectators to objectify themselves externally 
from a given image, which, in turn, delivers a deictic proclamation from a 
particular location, in a specific moment in time. The dynamic caused by 
perspective in paintings allows viewers to be positioned in places in which 
they have never physically been; in other words, presence and absence are 
intertwined in an indissoluble relationship (10). This relationship estab- 
lished from a vanishing point corroborates my investigation of ekphrasis 
inspired by architecture. 

In the realm of the visual arts, the notion of perspective has changed in 
the wake of the (still controversial) work of Edwin Panofsky, Perspective as 
Symbolic Form (1997 [1937]). James Elkins explains how, differently from 
our current comprehension of the notion of perspective, Renaissance art- 
ists and writers conceived of “unchangeable” concepts, such as the idea of 
pictures being “in perspective”. According to him, the greatest shift in the 
understanding of perspective was departing from its understanding “as a 
mute method, a practical subset of geometry” and adhering to “a power- 
ful concept for ordering our perception and accounting for our subjectiv- 
ity” (Elkins 1994: xi). Thus, when it comes to this favoring of its 
metaphorical understanding and forgetting it as a practice, there are 


70 M. VIEIRA 


inevitable consequences to the way we perceive it in painting, philosophy, 
literary studies (xi), and architecture. That is, for him, “perspective directs 
our eyes and orders our thoughts [...] it seems to control not only what I 
see, but how I see and how I describe what I see” (Elkins 1994: 212). The 
difficulties of thinking through and around a certain viewpoint arise 
because, at least for the visual arts, perspective is mostly self-confined. For 
Elkins, “the modern concept of perspective is schizophrenic, [in a way], 
because it has two incompatible aspects”. The first is the “formal, rigor- 
ously defined branch of [math]: the perspective that we trace from 
Brunelleschi’s experiments through the latest software”, or today’s under- 
standing of the term perspective inside architectural studies. The second 
aspect “means a great deal, from subjectivity to eternity, and it is to be 
found virtually everywhere, from philosophy texts to political speeches”. 
Elkins calls this second aspect, not derived from an equation or a graph, 
the metaphorical perspective. He explains that this “is ‘our’ perspective, 
the one we think about, and the one that describes how we view the world 
and constitute ourselves as viewing subjects” (6). Consequently, this claim 
for an exclusive individual point of view, as proposed by visual art theo- 
rists, is likely to engage with the notion of focalization as articulated by 
narratology studies. 

The narratologist Gérard Genette (1972) acknowledged the terms 
“perspective” and “point of view” as synonyms and coined the term 
“focalization” as a replacement for both. Grounded on the premises sug- 
gested by Genette, Tamar Yacobi (2002) points out that perspective has 
revolutionized both the theory and the practice of European painting 
since the Renaissance, activating lines of communication that bring artist 
and spectator closer together. Likewise, both Liliane Louvel (2010) and 
Stephanie Glaser (2014) consider the narrator's focalization in ekphrastic 
passages dealing with architecture. According to Louvel, by offering both 
aesthetic and artistic context, the visual information brings additional clar- 
ity to the literary text. She states that, differently from painting, the pres- 
ence of architecture in literature deals with the passage from architecture's 
three-dimensional space to the two dimensions of the literary object. She 
illustrates this argument with the opening of Wuthering Heights, in which 
a trompe-locil effect reveals a labyrinthine space concealing the plot’s 
secrets (Louvel 2010: 118-120). Accordingly, the house’s facade, as focal- 
ized by Lockwood's character, is treated by Louvel as a historized page of 
its residents’ lives. Glaser in turn bases her proposal on the notion of focal- 
ization suggested by Genette and revisited by Mieke Bal.!? For Glaser, the 
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ekphrasis of an architectural site may deepen the narrative or become 
attached to it on a thematic level (Glaser 2014: 13-15). For me, this pro- 
cess of verbally presenting architecture is likely to engage the narrator's 
gaze in a way that implies not only the observer's physical position in rela- 
tion to the architectural space—the literary persona’s focalization—but 
also their eye movement. 

Aiming at incorporating focalization and other types of perspective 
present in spatial descriptions from nineteenth- and twentieth-century 
novels (nouveau roman included), the narratologist Monika Fludernik 
(2014) developed a typology based on a model for the triple distinction of 
perspective, used by Holly Taylor and Barbara Tversky in their studies of 
cognitive linguistics (1996). According to these authors, when describing 
a space or a place, a speaker can guide the addressee on a mental tour from 
three different perspectives: (a) the survey, (b) the gaze, and (c) the route. 
The survey perspective is neutral, like an aerial view. The use of deictic 
terms is avoided in this type of description, which favors the universal lexi- 
con of orientation (North, South, East, West). This, according to Taylor 
and Tversky, is most common in literary descriptions, tourism guides, and 
textbooks. From the narrator's perspective, or from a route perspective, 
the observer is inserted into the narrated space. In type (b), the descrip- 
tion, the observer stands on a fixed location and makes use primarily of 
stative verbs. Finally, in type (c), by favoring action verbs, the observer 
traverses the environment (Taylor and Tversky 1996: 371—376). With the 
goal of broadening this model within the context of narratology, Fludernik 
combines types (b) and (c) into a single category and creates subdivisions 
in order to demonstrate that spatial descriptions are not necessarily con- 
fined to the extradiegetic level (Fludernik 2014: 472-474). In sum, the 
focalization of the (ekphrastic) passages will fill functional gaps within the 
plot of a literary text, and the many facets of perspective will be revealed 
through literary descriptions of spaces, places, and buildings—that is, of 
architectural ekphrasis. 

Although focalization and perspectivism do not necessarily coincide, 
both bring about the relevance of the observer's consciousness and 
embodiment as the organizer of verbal compositions. I therefore propose 
to add the notion of embodiment and to stretch the limits of perspective 
and focalization as girders in determining exactly which architectural 
modes may be successfully revealed by works of fiction. 
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3.4 THE DECK, OR ARCHITECTURE’S MEDIAL TRAITS 


This realization that architecture's modalities and its design process may 
be conveyed by words led me to the understanding of the actual modes 
that may be feasibly transmediated to literature. The parameters were set 
by combining Ellestróm's proposed model for studying the transfer of 
media characteristics between dissimilar media to the notions of embodi- 
ment and perspective, namely (a) the appeal to the senses, likely to be 
revealed through the user's bodily interaction with the building; (b) the 
employed lexis, based on the materiality revealed at the physical level, 
through materials such as concrete and glass, and the virtual level, through 
what lies in both clients’ and architects? minds; (c) language functions; (d) 
citations of architects’ and edifices’ names and titles; (e) meaningful repe- 
titions; (f) possible structural similarities and the importance of diagram- 
matic forms of representation; (g) verbal negotiation in assigning meaning 
over the course of the architectural process; (h) the indissoluble space- 
time relation in man-made environments, marked by deictic indicators; (i) 
the three-dimensionality of space, as proven by the ratios between height, 
length, and depth; (j) the role of affordance and ergonomy in relation to 
aesthetics; (k) the dynamics of perspective, focalization of the excerpt in 
question, and the literary persona's position in relation to the vanishing 
point, which will guide the receiver's eyes; and, finally, (1) the intercrossing 
of all the aforementioned evidence with historical, cultural, and social con- 
textualization of the architectural process as well as the positioning of cli- 
ent, architect, and final users. These parameters provided the deck for my 
proposal of an interpretative model for architectural ekphrasis that encom- 
passes four horizontal and non-hierarchical typologies: technical, sym- 
bolic, contemplative, and performative.!* 

The inherent potential to store information throughout an architec- 
tural process, from its inception to the edifice per se, guarantees its status 
as a medium. Developed in a dynamic communicative process, each stage 
of its design is fulfilled through specific qualified submedia (sketches, 
plans, blueprints, mockups, CG renderings, and 3D models), all of which 
require prior technical knowledge. In this case, both semiotic and material 
modalities play equally important roles within the transfer of medial traits. 
Conversely, when the beholder's cultural background knowledge is at 
stake, the semiotic modality prevails. I understand these two types of pro- 
cedures as technical and symbolic, respectively. 
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There are cases in literature wherein characters contemplate architec- 
tural environments, buildings, or even urban landscapes from a specific 
point of view, without the focalizing agent physically interacting with the 
focalized architectural site. In such cases, the transmediation of architec- 
ture may borrow medial traits from other media, such as painting, photog- 
raphy, or sculpture. Here the notion of perspective plays a highly relevant 
role, since it is the focalizer’s relation to the architectural site that will 
determine its contemplative nature. Architecture is treated as a two- 
dimensional medial configuration in cases where the focalizing agents 
behave as if they were in an art gallery. Meanwhile, when an architectural 
site is admired in its three-dimensional totality—as suggested by Aillaud, 
Rasmussen, and Kattenbelt—the focalizer behaves as if in a theatrical play 
and the edifice is likewise treated as a stage. Since embodiment plays a 
crucial role in the understanding of architecture as both process and media 
product, one may identify passages in which the observer, be it the narra- 
tor or a character, takes full advantage of the architectural medialities pre- 
sented thus far. In such cases, all modalities, along with embodiment, have 
an impact on the characters’ performative actions while touring an archi- 
tectural site—going up and down, in, out, and around it. The reader 
ceases to be a mere spectator and becomes involved in a virtual perfor- 
mance. In sum, since only a limited number of architecture’s medial traits 
can be satisfactorily transmediated by literature, the model here proposed 
by Ellestróm was crucial not only to delimit its modality modes, but mainly 
to bridge architecture to literature. 


NOTES 


1. Original text: *o ponto de encontro das ciéncias exatas e do humanismo” 
(Biselli apud Penna 2005: np). Unless otherwise stated, all translations are 
my own in collaboration with Maria Viana. 

2. Original text: “o espaco-síntese de toda a cultura? (Penna 2005: np). 

3. Original text: “arte e técnica de organizar espaços e criar ambientes para 
abrigar os diversos tipos de atividades humanas, visando também a deter- 
minada intenção plastica” (Houaiss and Villar 2009: 186). This definition 
is from a Brazilian Portuguese dictionary. I tried to find a similar definition 
in an English dictionary, but since the closest I got was *the art and prac- 
tice of planning and designing buildings" (Longman Dictionary 1995: 
56), I chose to translate the more thorough definition that expresses what 
I have in mind. 
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10. 


ll. 


12. 


13. 


. Patrik Schumacher (1961-) is the director of the firm founded by renowned 


architect Zara Hadid (1950-2016). 


. For more on architectural ekphrasis, see Vieira (2020, in press). 
. Please note that self-buildings and unplanned construction “on the spot” 


do not necessarily undergo the sort of professionalized architectural pro- 
cess on which my investigation is grounded. 


. These would traditionally mean sketches and presentation drawings— 


however, given the current predominance of architectural software, digital 
correspondents have now substituted the handmade graphic submedia that 
previously constituted this stage. 


. These submedia are analogous to the playscript of a theatrical production, 


since it can also be understood individually. Often, the computer graphic 
renderings, or three-dimensional models, mark the end of the process in 
the development of an architectural work, as in schools of architecture, or 
when the client terminates a contract due to financial limitations. Even if 
the process continues, this is often the point where the architect’s work 
ends. The architect can continue to follow the process during its technical 
stage, usually carried out with the aid of engineers, or during construction, 
but it is not obligatory. 


. These are initially developed according to local norms and legislation and, 


once approved by the due authorities, reworked to account for issues of 
engineering. 

Philippe Hamon (1999) proposes a model to investigate the mutual fasci- 
nation regarding the use of architecture in nineteenth-century French lit- 
erature. In this model, he establishes four reciprocal aspects—namely 
metalinguistic, semiotic, architectural representation, and historical 
context. 

The term affordance describes a given object's potential as suggested by its 
configuration, planned beforehand by a designer and manifested when an 
agent operates it in accordance with its structure. For example, a round 
doorknob encourages the agent to turn it. For more on this, see 
Gibson (1986). 

In the age of digital technology, the discussion still remains technical. With 
the introduction of computer tools at the end of the twentieth century, 
manual graphic representation became digital. Computer resources and 
software used in the elaboration of three-dimensional models, which have 
replaced artistic perspective drawings, may now produce architectural ren- 
derings so realistic that viewers are often unable to distinguish them from 
“real” (i.e. photographic) images. 

Gérard Genette (1972) proposes three types of focalization: zero, internal, 
and external. Mieke Bal (1985) revisits this categorization and suggests a 
binary system, composed only of the internal and external focalizations. 
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According to her, the type zero must be combined with the external, since 
it speaks not of the observer, but of the observed object. 

14. This model is more thoroughly presented in my doctoral dissertation 
(Vieira 2016). Its core has been compiled in Vieira (in press). 
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4.] INTRODUCTION 


The understanding that all media are multimodal and intermedial in some 
sense and share basic characteristics among them is now fully accepted by 
scholars at a time when studies focus on understanding the interaction 
between them. Comics are one of these originally intermedial arts, since 
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meaning is constructed through the combination of several media, verbal 
and nonverbal. This is done through different qualified media and their 
modalities, such as the graphic image.’ One of the characteristics of the 
perception of comics as qualified media? occurs quite simultaneously with 
their mediation. The combination of verbal and nonverbal languages 
found in comics, associated with the structure of the frames, is recognized 
even before the reading takes place (by word or image in this sense). This 
is how we associate the sequential combination of verbal language and 
graphic image immediately as comics, although comics media products 
may differ greatly in their qualities and functions. The same cannot be 
said about verbal literary narratives, for example: in this case, we cannot 
define merely at a glance whether we are looking at a novel, a tale, or 
youth literature. With ballet, a quick look at the ballerina's tutu is enough 
for us to recognize the genre. However, as soon as the dancer begins the 
movements on the stage, this dancer can be performing a role in a play or 
even street dance. 

In this chapter, our interest is not to analyze this media combination in 
comics fer se, but rather, to look into the strategies used by sequential art 
that prompt the term “journalism comics”, that is, how its modalities and 
modes differ from "fiction comics" in order to refer to the extradiegetic 
world. When analyzing the difference that seems most basic to us in com- 
paring these two sequential art forms—the literary and the journalistic— 
the distinction between “fiction” and “history” immediately emerges. 
Another issue that arises from this comparison is the distinctions and 
similarities between biography comics and journalism comics. Although 
biography can be understood as a journalistic genre, the distinctions 
between these two at times can highlight the notion that it is part of the 
journalistic strategies. In this sense, the analysis of media products pro- 
posed here focuses on the strategies of indexicality established by journal- 
ism comics, which puts the semiotic modality of comics in perspective. 

Our purpose is neither to analyze the conditio sine qua non of com- 
ics, nor to add complexity to basic concepts that are still the object of 
reflection and controversy (although they have already been settled in 
different areas), but to analyze the ways in which journalism comics 
and biography comics create indexicality through intermedial rela- 
tions. We have selected some basic terminology to aid in the under- 
standing of the ideas to which the concepts refer. Some additional 
clarifications are explained in footnotes. Lars Ellestróm's concepts of 
analysis will be discussed in section 11.3. 
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4.2 A SHORT History oF COMICS AS A QUALIFIED 
MEDIA TYPE 


The tactic of narrating facts by means of successive drawings, which 
requires the sequential reading of images, is quite ancient in human his- 
tory. To illustrate, one can refer to cave paintings in prehistory, ancient 
Egyptian hieroglyphics, mosaics, frescoes, and tapestries from the Middle 
Ages. As a specific example, one is reminded that in 1064, the Bayeux 
Tapestry told the Norman conquest of England in 58 scenes embroidered 
on 70 meters of fabric. Also, the Bible Pauperum, directed to illiterate 
Christians, preached religious faith from a series of woodcuts depicting 
scenes from the Old and New Testaments, focusing mainly on the drama- 
tization of the life of Christ. And the Harlot’s Progress series of paintings, 
produced by William Hogarth in 1730, consisted of six sequential paint- 
ings chronologically telling the life story of a prostitute in London. These 
classic examples of sequential narratives can be considered the precursors 
of comic books, which would have been recognized early in the print 
works of writer and illustrator Rodolphe Tópffer. Considered by many to 
be the great pioneer of comics—with the creation of M. Vieux Bois in 
1827— Tópffer framed his work in the form ofa caricatured novel, seeking 
to distance himself from the role of book illustrator. Tópffer's pioneer 
work was one of the first integrations between literature and graphic imag- 
ery to go beyond the merely illustrative function that nonverbal images 
had in their interaction with the word in literary works. 

This hybrid nature of comics, derived from media combinations of ver- 
bal and nonverbal languages, leads to an increase in the complexity of their 
conceptual definition. Eisner emphasizes that the art of comics is an act of 
aesthetic perception and intellectual effort, since the reader needs to exer- 
cise verbal and nonverbal visual interpretative skills, considering that the 
rules of painting (perspective, symmetry, brush stroke) and literature 
(grammar, plot, syntax), for example, mutually overlap (Eisner 1995a: 8). 
The author believes that both functions are irrevocably intertwined, since 
“sequential art is the act of weaving a fabric? (1995a: 122). Although 
McCloud defines comics as “juxtaposed pictorial and other images in 
deliberate sequence, intended to convey information and/or produce an 
aesthetic response in the viewer” (1994: 9), it is clear that the idea of 
sequential art is the only definition we will need to understand comics. 
Still, for many authors, such as Chinen (2011), the mere fact that the defi- 
nition of sequential art is also applied to cinema is enough for it to be 
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refuted in comics. When we connect the previous statement to Ellestróm's 
ideas (2020), we come to the idea that the spatiotemporal modality is 
narrative-sequential, though there are differences among the other modal- 
ities that make these media different. 

This conceptual vagueness is justified; capable of assimilating and 
transforming the languages of various media, such as cinema, literature, 
theater, and painting, comics also adapt easily to new technologies, and 
regularly adjust their method of production and meaning to available 
creative platforms. 'This is the case of their specific relationship with jour- 
nalism, since it was on the pages of newspapers that comics established 
themselves as a media product, before finding their narrative space and 
consolidating themselves as recognized qualified media. 

As far back as the mid-nineteenth century, newspapers were known for 
their pages with informative and opinionated material in the form of large 
blocks of text. Even before this period, in 1796, with the creation of the 
lithographic printing system, readers could access the news combined with 
visual elements. Created by Johann Alois Senefelder, lithography simpli- 
fied paper printing through a planographic process and made it easier to 
add drawings, thus changing the way newspapers were read. This *visual 
addition", seen as an informative complement, enabled the emergence of 
periodicals, especially in the post-revolutionary France of 1830, which had 
their greatest attraction in caricature. According to Bruchard, in her pref- 
ace to Daumier's book (Bruchard 1995: 4): 


Caricature was part of a movement in which a whole generation of artists, 
conveyed by an increasingly agile press, were able to establish a way of 
observing, describing and judging what was trendy in the modern sense of 
the term, with the scope, impact and media developments that it maintains 
until today.? 


The idea of caricature was based on the publication of prints containing 
political satire and social criticism, mainly targeting local rulers and politi- 
cians and their relations with the community. Honoré Daumier was one of 
the biggest names in caricaturism, devoting himself to political caricature 
and the defense of liberal ideals (Bruchard 1995: 4). His drawing 
“Gargantua”, which criticized King Louis-Philippe, “made him spend six 
months in detention, first in the Sainte-Pélagie prison, then at Dr. Pinel’s 
nursing home”* (Bruchard 1995: 4). The author Philipon published a 
sequence of drawings in the 1834 newspaper Le Charivari that also 
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criticized the king in an iconic way: the king's face slowly turns into a 
pear—the French word poire meaning pear, but also meaning foolish. 
Fined for the publication, Philipon ironically repaid his debt by selling the 
same print separately (Bruchard 1995: 9). 

As for the lack of resources for the visual recording of facts, since pho- 
tography was still incipient," illustrations became an indispensable narra- 
tive and informative element of the press. This is why the idea of illustrated 
journalism became popular in the most diverse segments: local news, big 
parties, social events, and mainly war coverage started to be narrated with 
the aid of illustration, though not necessarily sequential. In the USA, the 
press began to cover actions such as the American Civil War in 1861 with 
illustrations that were independent from opinions regarding the journalis- 
tic event—they merely portrayed different moments of action, such as 
open battle scenes and the personification of heroes. In Europe, so-called 
special correspondents gained recognition, now covering illustrated wars 
and world conflicts. In this context, great names in illustrated journalism 
emerged, such as William Simpson (Ilustrated London News), Sydney 
Prior Hall (The Graphic), and Melton Prior (Illustrated London News). In 
Brazil, the journalist Angelo Agostini was already developing modern con- 
cepts of the factual, documentary, and investigative idea of journalistic 
reporting with works such as *Scenas da Escravidão” (Slavery Scenes), 
published in Revista Illustrada in 1886. His historical comic books criti- 
cally portrayed the brutality of slavery and denounced, in a crude, docu- 
mentary form, the reality of punishment, torture, and murder committed 
against black slaves on farms, shortly before the signing of Lei Áurea, the 
Golden Law, in 1888. However, in the late nineteenth century, the 
improvement of photographic fixation, reproduction, and printing tech- 
niques was essential for photography to gain the pages of newspapers as an 
illustrative and informative feature of reports. The use of photography 
then provokes “a crisis of readaptation in the universe of representational 
art, ‘deprived’ of realism by another realism” (Sousa 2000: 24). 


For photojournalism, conquering movement proved to be of vital impor- 
tance, since it enabled one to ‘freeze’ action, impress it in an almost real- 
time image, capture the unforeseen, gain control of the snapshot and, with 
it, hint at the idea of truth: that which was thus captured would be true; the 
image would not lie (note, however, that although the snapshot allows for 
more ‘sincere’ and spontaneous photographic representations, the photo- 
graphs are nonetheless representations). (Sousa 2000: 29-30) 
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At this point, photography began to occupy a privileged spot in jour- 
nalism due to its indexical and iconic capacity to represent reality. This 
implied that the photographic representation would now be seen as an 
imaginary documentation of the world through a “realism” that painting 
and drawing could not portray (Sousa 2000: 33). Furthermore, *it also 
benefited from the notions of ‘proof’, ‘testimony’ and ‘truth’, which at 
the time were deeply associated to photography, deemed as a ‘mirror of 
the real’” (Sousa 2000: 33). Curiously, during a photographic exhibition 
in Paris in 1855, photographer Franz Hamfstangel exhibited, for the first 
time, retouched proofs of negatives, raising questions about the very 
essence of journalistic photography. According to Sousa, “if Hamfstangel 
invented the retouching of the negative, it also opened the door to the 
manipulation of the photographic image by trickery” (2000: 32). 

With photography occupying the space of the illustration of reality in 
journalism at the time, the emerging media of comics adjusted to newspa- 
pers as a product of entertainment, based on fictional narratives. More 
specifically, comics sprang from a journalistic rivalry between William 
Randolph Hearst, owner of The New York Journal, and Joseph Pulitzer, 
owner of The New York World, who were fiercely vying for the market’s 
growing advertising dollars. Considered by most critics to be the first 
comic book, Richard Felton Outcault’s Hogan’s Alley series, released in 
1896 in The New York World, portrayed the daily lives of various exotic 
and caricatured characters living in a New York alley. Among them there 
was the boy Mickey Dugan, who became known as Yellow Kid? because he 
wore a yellow nightshirt on which criticism was written. The series estab- 
lished the new cultural and commercial advertising production of comics. 

Comics then began to dispute available spaces in the colorful Sunday 
newspaper supplements, which were mostly a hit among children. In 
1907, author Bud Fisher proposed the creation of a *strip-shaped" comic 
with a generally horizontal fixed format with few frames, facilitating both 
simultaneous production and distribution to various newspapers. With 
daily delivery and format limitation, the comic strips began to focus on the 
theme of humor, portraying comical situations that resembled jokes with 
generally unpredictable outcomes and simpler themes. This funny style of 
comics was so strong and influential that the expression comics is used to 
this day to specify any kind of comics.? Parallel to comic strips as entertain- 
ment, all kinds of newspaper cartoons which assimilated the ideas of the 
old caricatures consolidated their role as political and social criticism, 
becoming tools of opinionated journalism and contestation. 
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With the emergence of a consumer comic book market, the most suc- 
cessful newspaper comic strips began to be compiled and printed in book 
form, then called comic books, the beginning of the form we know today. 
In 1938, with the publication of Superman stories, the golden age of 
superhero comics was born, ignited by a society crushed by the horrors of 
World War II and eager for examples of heroes. As Eco (1972: 107) states, 
quoting Superman's example, *the hero equipped with powers superior to 
those of the common man has been a constant in people's imagination— 
from Hercules to Siegfried, from Roland to Pantagruel all the way to 
Peter Pan”. 

In the 1950s, superhero comics, which were once examples of the val- 
ues of unity, freedom, and overcoming difficulties, were challenged by 
rigid and distorted moral concepts at the time. It was in this context that 
the ideas of German psychiatrist Frederic Wertham, author of the book 
Seduction of the Innocents, were published and gained the spotlight in the 
USA in 1954, warning of a possible danger coming from comic books, 
and accusing them of stimulating juvenile delinquency. The destructive 
power of Wertham's book was immense and ultimately influenced an 
aggressive comic book censorship campaign, which eventually became the 
target of congressional investigation. The result was the creation of a 
comic self-censorship code—the Comics Code Authority (CCA)—which 
regulated the industry and wiped out numerous outstanding magazine 
titles that did not fit the moral standards of the time. 

It was in this scenario that underground-styled comics stood out, a 
counterculture movement aiming at criticizing the values of the time. The 
main themes of the movement were sex, drugs, rebellion, and the Vietnam 
War. The underground comics intended to establish an anarchic and criti- 
cal style, resulting in the development of misleading and irreverent charac- 
ters and the creation of authorial and challenging stories based on the 
lifestyle of their authors. 

In general, the printed newspaper never proved to be a receptive space 
for comics that featured longer and authoritative narratives: the preference 
was usually for the objectivity of the event in news production, which 
pointed to a journalistic need for informative material with daily or weekly 
update. Even when there was the possibility of developing longer and 
more investigative stories, comics were not a production priority due to 
the time required for production, delays, and the costs involved. 

It was with the rise of underground-style comics reviews, usually pub- 
lished in magazines and books, that nonfiction author comic books were 
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consolidated, pointing toward the appropriation of the real inherent in 
history, journalism, and biography. Turning to biographical and autobio- 
graphical stories, comics began to concern themselves with portraying 
daily life drawn from the experiences of the author or the observed char- 
acter, in the form of personal stories, family stories, or common dramas 
often intertwined with great facts of history. In this context, involvement 
with the characters’ psychological, sentimental, and motivational issues 
emerges—illustrated by memory flashbacks—whose subjective nature 
reveals interpretations and opinions made in a very personal and particular 
way. The presence of this aspect in comics media products can be analyzed 
in different ways. 

In comics, the very idea of the biographical narrative establishing itself 
as an element of social and cultural representation—while also acting as 
historical document and rescuing the past and contextual analysis of an 
era—eventually strains the limits of journalistic narrative. Since comics use 
iconic representations of reality, they seek to legitimize situations and con- 
texts through indexical representations, such as graphic images of docu- 
ments or factual situations, highlighting important and recognizable 
details to create models of reality representation. 

This is how biographical comics often use journalistic resources to doc- 
ument this reality, trying to ensure the credibility of the reader, who needs 
to make a plausible connection between verbal and nonverbal elements 
and accept them as valid representations of a factual narrative. Thus, many 
comic book authors use representations of photographic images, excerpts 
from newspapers and official documents, as well as recurring methods of 
researching journalistic information, such as interviewing and investigat- 
ing facts from different sources. This is the case of Will Eisner, Art 
Spiegelman, and Joe Sacco, authors who subtly or directly incorporated 
these features into their narrative styles and consolidated a path that would 
later be known as comic journalism. 

Will Eisner was one of the precursors in the creation of biographical 
comics, portraying his life in New York City, especially in the autobio- 
graphical works The Dreamer (1986)—which presents a faithful portrayal 
of the universe of comics creation during the 1930s—and To the Heart of 
the Storm (2000b)—which portrays his life trajectory, especially his 
upbringing in a poor US environment in the early twentieth century. In 
this work, Eisner uses a continuous graphic feature to show characters 
reading the newspapers of the time, whose headlines project the reader in 
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time and space, chronologically showing historical facts pertinent to 
the plot. 

The urban theme of his biographical stories is also present in comics 
such as A Life Force (2001), Dropsie Avenue (1995b), and New York City 
Life in the Big City (2006). In these works, Eisner connects the reality of 
his characters, who intersect in various stories, using real places and facts 
that appeared on the pages of the city’s newspapers. For example, in 
Invisible People, according to Kitchen (2009), Eisner was inspired by 
Carolyn Lamboly’s true story, found in a New York Times report about her 
suicide in 1990 and the motivations for her desperate gesture. The book 
A Contract with God (Eisner 2000a), which popularized the term “graphic 
novel"? and was one of the stories responsible for the recognition and 
consolidation of comics life narratives, based the creation of its controver- 
sial characters on real experiences, according to the author: 


In this book, I have attempted to create a narrative that deals with intimate 
themes. In the four stories, housed in a tenement, I undertook to draw on 
memory culled from my own experiences and that of my contemporaries. I 
have tried to tell how it was in a comer of America that is still to be revisited. 
The people and events in these narratives, while compounded from recall, 
arc things which I would have you accept as real. Obviously in the creation, 
names and faces were rearranged. (Eisner 2000a: 8) 


In Maus. The Story of a Survivor (2003), author Art Spiegelman fuses 
the boundaries between autobiography, biography, and journalism by 
interviewing his own father, in an attempt to reveal his family’s martyrdom 
during World War II. In reliving the horrors of the Nazi holocaust, 
Spiegelman eventually reconstructs fragmented identities of Jewish people 
destroyed by Nazi ideals, picturing each nationality in the form of an ani- 
mal. Considered one of the most important comic books ever, Maus 
became a reference by winning the esteemed Pulitzer Prize in 1992. 

However, it was journalist and comic artist Joe Sacco who really com- 
bined comic book narrative and features of the story as an investigative 
narrator, figuratively represented by the reporter’s graphic image. 
Portraying himself as a reporter in Palestine (2003), Sacco enters the 
Israeli- Palestinian combat zone to hear the voices of war victims, usually 
conditioned to statistics in history books and the world press itself. With a 
closer look at contextual information, the author shows a subjective jour- 
nalistic report, which provides a “face” to the invisible Palestinian 
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characters,'° until then doomed to the forced silencing of identity, always 
associated collectively with terrorism. 

Sacco's work, which includes many graphic novels about wars around 
the world, is based on his journalistic investigations, withdrawing from 
interviews and sources of information, in which the author portrays him- 
self and sees his function as a journalistic narrator. By contextualizing facts 
and journalistic and historical data, Sacco assumes a documentary style 
without giving up his point of view. Subjectively, the author details his 
actions and his creative processes: narrating his efforts as a journalist, visu- 
alizing his difficulties in obtaining testimonials, and telling his own story 
within a conflict that involves different and contradictory positions. 


4.3 | MEDIA REPRESENTATION AND TRANSMEDIATION 
IN JOURNALISM COMICS AND BIOGRAPHY COMICS 


Some boundaries regarding the concepts of media and intermediality are 
important for this article, although it is quite specific in its objective: to 
analyze the semiotic modality of journalistic and biographical comics as a 
way to understand the strategies of building iconic and indexical represen- 
tations through graphic images. Before moving on to the analysis, we will 
investigate some of these boundaries. What we understand as media and 
media modalities are clearly circumscribed in Lars Ellestróm's “The 
modalities of media II" in this volume (2020). However, it is essential to 
elucidate how our analysis appropriates these concepts. 

First ofall, based on the understanding that sequential art is a combina- 
tion of several media—verbal and nonverbal—this chapter takes *journal- 
ism comics" and *biography comics? as qualified media types in the terms 
defined by Ellestróm; he states that “[q]ualified media types are simply 
categories of media products grounded not only on basic media modality 
modes but further qualified" (2020: 55) Although the distinction between 
basic media types and qualified media types is not absolute since, as 
Ellestróm points out, they are dependent on historical and cultural criteria 
as well as communicative purposes, it is not difficult to understand this 
differentiation related to the media products analyzed here. However, two 
points need to be outlined. 

The first concerns the fact that when we compare “journalism comics” 
and “biography comics” with “fiction comics”, similarities and differences 
stand out. The similarities arise from the comparison between the 
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material, sensorial, and spatiotemporal modalities of these media types: 
they are all solid, visual, two-dimensional media products produced from 
static images—and they all belong to the “comic book” medium type. 
Moreover, they are all narrative, which implies a certain sequential struc- 
ture that is fundamental for comics, differentiating them from, for exam- 
ple, illustrated books. Apart from dissimilarities in themes and style, their 
differences lie in their different modes in the semiotic modality. Journalism 
comics and biography comics both differ from fiction comics in terms of 
the semiotic modality; whereas fiction comics generally emphasize the 
symbolic and iconic mode, journalism and biography comics largely rein- 
force the iconic and the indexical modes. In accordance with Ellestróm 
(2020), it must be emphasized that these are categories of analysis; when 
described from their different semiotic modes, media products show 
themselves as different types of qualified media, although their signifi- 
cance depends on the reader and even on the contract established with 
the text. 

The second point that cannot be disregarded concerns precisely 
that contract. When published, the categorization intended by the 
editors and authors for their works—designated by the qualitative 
“journalistic”, “biographical”, or “fictional”—evokes a certain repre- 
sentation on the part of the readers. In journalism and biography 
comics, the classification in the catalog sheet causes the mediation of 
objects—people, things, events, and media—by both verbal and non- 
verbal media, to build indexical representations of the objects of real- 
ity. This means that, although semiotic modes are not absolutely 
distinguishable—the symbolic, the iconic, and the indexical work in 
collaboration—one of the semiotic functions is highlighted by the 
intention of the text, whether journalistic or biographical (or fic- 
tional). This is an example of what Ellestróm, in *The modalities of 
media II" (2020), points out as the communicative function of semi- 
otic modes: although the semiotic modes are based on presemiotic 
modes, it is only in semiosis—representation of meaning—that mean- 
ing is constructed. 

Thus, our analyses in this article focus on the semiotic modes of media 
products, whose symbolic and iconic representations of graphic images 
simulate the indexicality of journalism and biography. Hence, we offer a 
semiotic interpretation of media products such as journalism comics and 
biography comics. Nevertheless, one pertinent detail places this analysis 
on the axis of intermediality studies: the fact that, to tell a journalistic 
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story, comics appropriate journalistic strategies. In other words, they rep- 
resent and transmediate the qualities of the media type journalism to sim- 
ulate its characteristic indexicality. This strategy is also typical of biography 
comics, but in a different way. Biography comics are more related to his- 
torical narratives, rendering the representation of documents and photo- 
graphs concerning people and their actions within a recognizably historical 
context the most important aspect. In this sense, the representation of 
media in biography comics acquires another meaning. 

The concept of representation here gains a second connotation: specifi- 
cally, the representations in which the object represented is another media 
type or another media product—an intermedial phenomenon. Ellestróm 
calls this “media representation”, which is one of the two forms of media 
transformation, or intermedial transmediality (the other one is 
transmediation): 


My first step toward the goal of forming a conceptual framework that facili- 
tates a detailed analysis of transfers of media characteristics among media is 
to establish a distinction between transmediatian (repeated representation 
of media traits, such as a children’s book that tells the same story as a com- 
puter game) and media representation (representation of another medium, 
such as a review that describes the performance of a piece of music). 
(Ellestróm 2014: 12) 


As we intend to show in this analysis, media representation is a funda- 
mental trait of journalism comics, since it is used to refer or allude to 
journalistic media through the presence of newspapers, television, com- 
puters, and cell phones being handled by characters (simple media repre- 
sentations) and to create more elaborate representations of specific media 
products or qualified media types (complex media representations). Based 
on Charles Sanders Peirce, Ellestróm distinguishes three types of represen- 
tation: deiction (iconic), deiction (indexical), and description (symbolic) 
(2020: 51). However, it is quite difficult to categorize a complex media 
representation in these terms, especiall when one medium imitates 
another. 

In complex media representations, the content of the quoted and some- 
times imitated medium is crucial to the representation of that medium. 
The strategy of representing and imitating other media is generally 
intended to point to the extradiegetic factual reality—producing indexi- 
cality. In the case of simple media representations, there is a certain 
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invocation of the very type of qualified medium—journalism. As for biog- 
raphy comics, the indexicality is shown more by the indexical representa- 
tion of people and events, signs that often appear “inside” other media, as 
we will demonstrate below. 

One of the characteristics of media representation is that the “frame” of 
the represented medium can be perceived; that is, we can view the repre- 
sented medium as a medium inside another, in some of its modalities. The 
graphic image of a printed newspaper on the table is easily recognized as 
the “printed newspaper” media type (which is not the case in transmedia- 
tion, when the frames become indistinct or disappear). A cell phone in the 
hand of a character in a movie can mean simply that he is a person living a 
common routine these days; however, it can also mean that he is sending 
someone a message. That message, if perceived, like any other media 
product, may be more or less relevant for understanding the story of 
the film. 

In complex media representations, however, represented media charac- 
teristics dissolve in the representing media product and thus insert details 
that are generally more important for meaning representation. The 
detailed description of a particular section of a newspaper, for example The 
Times, is not just meant to represent any newspaper as a qualified media 
type, but a specific media product, invoking idiosyncratic characteristics 
typical of that product, such as the ideology of its editorial line. Such a 
representation is probably simultaneously iconic, indexical, and symbolic. 
Another example: in Emmerich's 2011 movie Anonymous, there are sev- 
eral simple representations of audiovisual media types. Theater, for exam- 
ple, is iconically represented by images of theater buildings and stages. 
Some representations are more indexical, as when the Globe Theatre is 
represented through what the reader perceives as real connections, and 
symbolic, as when there is a character in costume, ready to act, or when a 
character says “theatre”. There are also simple media representations in 
references to Shakespeare's play titles. Furthermore, there are complex 
representations of Shakespeare plays in film, setting in motion the iconic, 
indexical, and symbolic modes simultaneously—in words, in costumes, in 
dialogues, in the interpretation of the characters, and in theater settings. 
In this way, the film appropriates features of Shakespeare's plays in vari- 
ous ways. 

The boundaries between complex media representation and transme- 
diation are sometimes subtle. A scene from a Shakespeare play being 
played in a movie also needs to adapt the play's dramatic text (which 
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becomes part of the script) to that scene (the scenic text that begins to 
integrate the movie). If it is completely “staged in” the movie, on a “filmed 
stage”, we can say that there is an iconic simple representation of both a 
qualified media type (Elizabethan theater) and a media product 
(Shakespeare's play )—as well as a transmediation of Shakespeare's play. 

Transmediation, according to Ellestróm (2013), happens when content 
is re-mediated. A play can be adapted for the cinema; a dramatic text may 
be performed on the stage; or it may be read aloud to an audience. In 
transmediation, it is not the previous media that appear, but their content, 
which is transmediated—adapted to the new media and their modes of 
mediation and representation of cognitive import. Given the limitations of 
comic book affordances, which can only adapt media types that somehow 
fit their characteristics—solid, visual, two-dimensional, made up of static 
images—media representations are more common than transmediations 
in comic books, especially considering the need to show the frame, that is, 
to make a reference also to the qualified media type “journalism” to simu- 
late its indexicality. In this sense, it is not enough to represent again the 
content of a particular television report; it must also be shown that the 
content was mediated by a television report and can be said to be journal- 
istic content. 

Based on our interpretation of Ellestróm's (2014) concepts of media 
representation and transmediation, we will now analyze three narratives of 
journalism comics: Footnotes in Gaza (Sacco 2009), I] mondo di Aisha 
(Bertotti 2008), and Le photographe (Guibert et al. 2003, 2004, 2006)— 
and three narratives of biography comics: Dotter of Her Father’s Eyes 
(Talbot and Talbot 2012), Persepolis (Satrapi 2004, 2005), and To the 
Heart of the Storm (Eisner 2000b). 


4.3.1 Simple Media Representations 


The comics analyzed range from simple representations of technical 
media to simple representations of specific media products. 
Interestingly, in view of this need to construct indexical representa- 
tions, generally the media products shown are not fictitious but “real”, 
often recognizable by the reader as literary books. Dotter of Her 
Father’s Eyes brings several examples of this, presenting different and 
varied simple media representations already in the paratexts, where 
there are graphic images of photographs that show some documents 
that, later in the narrative, will be represented again, this time by 
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illustration. An indication of this is that these photographic images 
state the initial tone of the qualified media type that is built there—in 
this case, a comics autobiography—signaling that the characters exist 
outside the narrative, that is, outside the images, through the copy of 
the photograph. Photographs are important documents in themselves, 
representing reality, and, despite the contemporary possibilities of 
manipulation, they are still the best way to build this contract with the 
reader. Documents, to the same extent, when presented through pho- 
tographic images, reinforce this “real existence” of the subjects por- 
trayed—falsifying a document is quite different from falsifying a 
photograph—and thus the narrative acquires “biographical” quality. 
In this sense, we can understand that even the illustration, which can 
be perceived as an iconic representation, becomes an indication—an 
indexical media representation. Perhaps we may also think that it is an 
iconic representation—the illustration of the photographic image—of 
an indexical representation—the photographic image (a graphic copy 
of the photograph). 

Still, in Dotter of Her Father’s Eyes, technical media such as television 
and radio, newspapers, magazines, and even comics serve to contextualize 
the historical moment and also create the atmosphere in which the charac- 
ters of the story live, people who have an intense relationship with the 
culture and the arts. However, in addition to representing the media as 
means and tools of communication, they are also used to indicate real situ- 
ations involving the factual context of the story. In the illustrated scene of 
the protagonist Mary’s conversation with her mother (Fig. 4.1), for exam- 
ple, we see a television set in the background, showing an image that 
stigmatized the Vietnam War: Eddie Adams’s “The Execution” photo- 
graph, winner of the Pulitzer Prize of 1969, which shows a summary exe- 
cution—a soldier shooting a prisoner’s head—in front of an NBC TV 
camera. The facts—the explosion, the war—are not crucial for the story 
(the narrated fable), but rather indicate truthfulness. Thus, the media 
product representation is not very important to the meaning of the work 
apart from referring to the extradiegetic reality, reinforcing the biographi- 
cal tone. The same can be said about the books and comics that characters 
in Dotter of Her Father’s Eyes have at hand. Most of them are simple rep- 
resentations of media, although they show titles and details of specific and 
real media products that aim to indicate the “real” readings of the bio- 
graphed characters. There is even an addition that, according to the narra- 
tor, does not correspond to the truth, since the illustrator added a book 
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I had more than a little teenage angst 
and took myself very seriously. 


DE QUO! 
As-Tu PEUR? JS WE 
WHAT ARE YOU NE 

Ú rRIGHTENEP OF? t 

| : 


convinced that 
I'm afraid of going insane. the world was about to end. 


Fig. 4.1 A television showing an image that stigmatized the Vietnam War 
(Talbot and Talbot, Dotter of Her Father’s Eyes, 2012: 56). Copyright Mary 
M. Talbot and Bryan Talbot. All rights reserved 


that is not one of the narrator’s favorites, but the illustrator’s. Thus, the 
representations of books cease to be mere illustrations of books in general 
(technical media) and become indications of works that help to define the 
characters as real people. 

In Persepolis, which is also an autobiography comic book, simple media 
representations pursue these same goals. When Satrapi’s mother, the 
protagonist-narrator, is photographed by a journalist, there is a painting 
showing the moment of the click, which is only evident as a photograph 
when shown “inside” the newspaper (Fig. 4.2). The fact that it is in a 
newspaper, and then in a magazine, demonstrates the problem of exposing 
the mother’s image in a protest, but the content of the media product 
matters as much as the representation of the media type newspaper and 
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AT ONE OF THE DEMONSTRATIONS, A GERMAN 
JOURNALIST TOOK A PHOTO OF MY MOTHER. 


1 WAS REALLY PROUD OF HER. HER PHOTO WAS 
PUBLISHED IN ALL THE EUROPEAN NEWSPAPERS, 
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Fig. 4.2 The moment the photograph is taken, shown “inside” the newspaper 
later (Satrapi, Persepolis: The Story a Childhood, 2004: 5). Copyright Marjane 
Satrapi. All rights reserved 


magazine. It is through the publication of this photograph that Satrapi 
finds out that she is the great-granddaughter of an ancient Persian king, an 
important fact for the history of the country. Delivering the reality of the 
outside world also occurs with representations of television, which often 
comes up to show specific TV programs and their content, reference that 
are also made verbally. The representation of media types as vehicles of 
information is important to signify the sphere of the Iranian government 
discourse delivery, when verbal media are required to refer to the content 
of that discourse. Literature representations are also important in Persepolis, 
as it is through literature that Satrapi tries to understand the world and her 
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role in social and political transformations. There are simple representa- 
tions of books—on the bookshelf, on the bed, on the nightstand, in 
Satrapi’s hands—whose titles and content are not noticeable, which high- 
lights both the technical book medium and the literature media type. 
There are also representations of specific media products, when the titles 
of the books are shown or when the character mentions them. These rep- 
resentations are often complex, as we will explain in the following section. 
The presence of pop culture and market consumerism, characteristic of 
Western culture, is often represented by images of posters and album cov- 
ers. Marjane Satrapi, the author, is a graphic illustrator and her relation- 
ship with drawing and painting is brought about by simple representations 
of the practice of drawing by the protagonist. 

In To the Heart of the Storm, as in the two previous autobiographies, the 
profession of the biographer is important. Although it is an autobiogra- 
phy, the purpose of the representation of the printed media here is similar 
to that of journalistic comics, since the biographer is the cartoonist Will 
Eisner who worked for newspapers—which explains the many representa- 
tions of the printed media type in the story. Being read or handled by the 
characters, for sale at newsstands, or thrown on the streets, the newspaper 
is more important as a qualified media type (as a vehicle of “truthfulness” ) 
than as a simple technical medium. However, for the biography, the news- 
paper also fulfills a chronological and informative function. It associates 
the protagonist’s life to his relationship with illustration, and his family’s 
life to the historical context—especially the economic factors surrounding 
his father’s business (specifically the crises occurring before and after 
World War I). The arts of illustration and painting are also part of the nar- 
rative, revealing many of their processes and techniques to illustrate this 
biographical datum. The idea of the representing reality in comics is also 
treated symbolically by painting: an artist destroys his painting when try- 
ing to scare off a fly that had landed on the canvas. However, the fly was 
just a drawing, intentionally created to engender confusion. 

The author’s autobiographical reflections start with the representation 
of photography by illustration. This reinforces the above-mentioned idea 
that biography comics build indexicality from a relationship between the 
graphic image of photography and real-life subjects and events. The atypi- 
cal frames of a photo album are used to reinforce the frame of the photo- 
graph itself as a media type that represents reality and thus triggers the 
memories of the character. 
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The simple representations of technical media have partly different 
meaning in Footnotes in Gaza, Il mondo de Aisha, and Le photographe. In 
these cases, in journalism comics, the iconic depictions of printed newspa- 
pers and television sets through illustrations serve as indices of activities to 
produce history and reinforce the importance of journalistic media types. 
In other words, while in biography comics the graphic representations of 
photographs of documents and people are vital for producing indexicality, 
in journalism comics, this indexicality is given by the representations of the 
technical media themselves. 

In Le photographe, a narrative about an expedition by Médecins Sans 
Frontières (MSF) by photographer Didier Lefèvre, the act of photograph- 
ing, shown by the photographic equipment, is often highlighted to repre- 
sent that it is a photographic report, even though it is constructed by 
illustration. In this case, the indication is given by the parallelism between 
the two graphic images, the illustration and the photographic image of 
that moment, or of the same people who were portrayed. Therefore, it is 
not a transmediation of a photographic image to illustration, but an iconic 
representation of other possible angles of that situation, as if the photog- 
rapher took photographs through the illustration. This idea is corrobo- 
rated by the fact that the photographer is often represented with a pen and 
a notepad in his hands, suggesting he was drawing the scene. When this is 
opposed to photographic images that show the same places and people in 
other perspectives, representation acquires indexical value. 

In Footnotes in Gaza, the journalist also appears photographing and 
writing on a notepad, and, in the same way, this reinforces the agency of 
journalism, inserting indexicality. This agency is also marked by the repre- 
sentation of the media type interview, performed through various strate- 
gies with the graphic image. From the beginning of the narrative, the 
protagonist, journalist Joe Sacco, stresses the importance of finding the 
people who have experienced the events he wants to report. Once he finds 
these people, he begins to interview them: the illustration shows this 
meeting, while the protagonist holds a tape recorder, notepad, and pen. 
Rather than illustrating a journalistic practice, the representation of the 
media interview is primordial to the narrative construction of the histori- 
cal memory of the war, as the details and contradictions they bring show 
the uncertainty of information on either side. The simultaneous graphic 
image of different opinions (sources) about the same event is an indexical 
representation of journalism, showing this characteristic of the journalist’s 
search for different versions; thus, the likelihood that journalism does not 
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bring up one single truth. Sacco, the narrator character, is present in each 
of these interviews in the illustration, usually recording the audio or taking 
notes in his notepad. 

The details of the graphical representation of the interviewees simulate 
the indexical representation of photography, making each one of them 
quite idiosyncratic, not to mention human, real, in their subjective charac- 
teristics. On the other hand, the graphic representation of the protagonist- 
narrator, journalist Joe Sacco, is quite iconic, almost caricatured, compared 
to his photographic representation, present in the digital version of the 
work and on the internet. This seems to be an attempt to highlight pre- 
cisely “the journalist” among the other characters, thus distinguishing him 
visually. 

In Il mondo de Aisha, photography acquires considerable importance as 
a legal resource as it receives, in the introductory pages, a subtitle: “A 
work inspired by Agnes Montanari’s travel reports”. Agnes Montanari is a 
photographic reporter who conducted interviews and photographs of 
Aisha and other women oppressed by Yemeni customs. Comics artist 
Hugo Bertotti adapts this story to comics"! by illustrating Agnes meeting 
Aisha during her job of shooting photos and interviewing people. In the 
comics paratext, there is a text by Agnes Montanari describing her meet- 
ing with Aisha and the other interviewees. The comics are divided into 
several stories and, in each of them, a photo of the character and the name 
introduce the narrative, showing their relationship with the extradiegetic 
reality—based on the photographic representation. At the end of I/ mondo 
de Aisha, the photos of all interviewees are shown. Both the camera and 
the act of photographing are very often represented. There are also repre- 
sentations of video footage and interviews for television programs by one 
of the characters. 


4.3.20 Complex Media Representations 


Some media representations become more complex as they enter the 
structure of the represented media product, producing meaning from the 
qualities of other media. Focusing on the work of the photographic 
reporter in I mondo de Aisha, for example, through the representation of 
the journalism agent, as well as in Footnotes in Gaza and Le photographe, 
we can see that indexicality is built by presenting the act of capturing real- 
ity, “as if” the reader were watching the news report through the “screen” 
of comics. This “as if", treated by Irina Rajewsky (2005, 2010) as 
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“intermedial reference”, can be understood, according to Ellestróm 
(2013), as a complex representation of media. Another type of simulation 
can be seen when Agnes Montanari sends Aisha her own photos: this send- 
ing of photos is symbolized in the verbal media by the word “send”, and 
illustrated in the construction of a photo trajectory that simulates a jour- 
ney through “internet waves" (Fig. 4.3).? The photographs, when 
received, are represented by the photographic images framed on Aisha's 
computer screen. Another complex representation of media in Aisha's 
world lies in the interweaving of the characters photographic images with 
the comics pictures, creating sequences in which both media types com- 
plement each other in a combination of media that makes it difficult to 
distinguish which represents the other. 

Although the literary work is authored by Didier Lefévre as a whole, Le 
photographe has the artistic participation of Emmanuel Guibert (script and 
drawings) and Frederic Lemercier (colors and montage/diagramming), 
which then produce a narrative from different techniques of photographic 
and cartoonistic composition that comes from different views and ways of 
expressing the world. Thus, we can imagine that the protagonist-narrator— 
the photographer— brings together a little of each of these arts, so much 
so that when he introduces himself to the reader at the beginning of the 
story, he does this with his iconic, illustrated image. It is only at the end of 
the story that his photographic image will appear addressing the reader. 
Prior to that, the only photographic image in which he appears promi- 
nently is in a symbolic representation, where he is photographed in the 
mirror without showing his face. Sometimes the act of photographing is 
illustrated to show the angle in which the photograph was taken, from 
which a photographic image is shown in the sequence (Fig. 4.4). Thus, 
formally, the whole story is a complex media representation, combining 
and blurring the limits between the arts. In this and other cases of complex 
media representation, it is possible to perceive the “work” that gives rise 
to the “image”, allowing a glimpse of metalanguage. 

In Footnotes in Gaza, Sacco portrays the daily work of journalism in the 
dangerous informational coverage of the war, illustrating several commu- 
nication professionals performing a vast array of practices. Conversely, he 
does not avoid criticizing his profession based on the choices made: jour- 
nalists seem not to care about people, but only about the sensationalism 
that the event can provoke, as in the representation of photographers who 
only focus on people in moments of pain and despair for more impacting 
images. Sacco's harsh criticism also applies to moments of rest: journalists 
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OBRIGADA PELAS LINDAS FOTOS, AGNES, E A IDEIA DE QUE FALA ME AGRADA... 
UMA REPORTAGEM QUE DB VOZ AS MISTERIOSAS MULHERES IEMENITAS... 
SAO TANTAS AS QUE TBM VONTADE DE CONTAR 
SUA HISTORIA, AINDA QUE MUITAS SÔ O FARAO 
DE MANEIRA ANÔNIMA. 

ESTAMOS TODOS, MULHERES É HOMENS, 
PRESOS NUM CIRCULO TRIBAL QUE SE ALIMENTA 
DE POBREZA E IGNORANCIA, QUE METE MEDO. 
MEU PAIS & ASSIM: 
ARCAICO, ARRUINADO, É MAGNIFICO. 
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Fig. 4.3 The sending of photos is symbolized by the word “send” and illustrated 
by a photo trajectory (Bertotti, I/ mondo di Aisha, 2008: 134). Copyright Ugo 
Bertotti. All rights reserved 
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Fig. 4.4 The angle in which the photograph shown later was taken (Guibert 
et al., Le photographe. Tome I, 2003: 47). Copyright Emmanuel Guibert, Didier 
Lefévre and Frédéric Lemercier. All rights reserved 


talk and exchange ideas about work and the nature of the information. 
The news seems to bring nothing new, because the war always seems to be 
the same and no one is impressed any longer. Since the “agenda had died” 
(Sacco 2009: 124), the “hope” is that a new war will break out to stimu- 
late journalists. Here, the complex representation of journalism media, at 
times when it represents journalists gathering in different situations, makes 
sense in its own story and even in its own essence, since it emerges from a 
footnote, that is, the event referred to in a footnote that did not become 
news because it was not covered by journalists. 

Another complex representation of media in Footnotes in Gaza is in the 
illustration of Moshe Dayan’s book Diary of the Sinai Campaign. Not 
only does the cover show it in minute details, but there are depictions of 
excerpts from the book. For example, there is a page with a map that not 
only shows a detail of Gaza’s historical geographical division, but also 
reveals the need for geographic knowledge for such a story. Graphic ele- 
ments widely used in Footnotes in Gaza, such as maps and other spatial 
geographical representations, are important since history is precisely about 
a war over a territory. The incident reported in Footnotes in Gaza is not in 
current history books and is only recalled by a brief citation in a footnote 
in a UN report,’ which states that Israeli soldiers had panicked and 
opened fire against the rushing crowd. However, the report issued by 
Israel is quite different, according to Sacco, who uses this document as a 
starting point for his investigation. 
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The illustrated representation of documents (official and unofficial, 
according to the narrator), for example, the illustration of a detail in the 
previously mentioned UN report, also shows the intention of making it 
seem that the strategies for producing a report are really journalistic. 
Although we can understand some of the above-mentioned media repre- 
sentations separately as simple, they also represent the production of jour- 
nalistic reports in complex ways, simulating a sort of “making of a 
documentary” rather than just narrating the story of the note about Gaza 
which is, as it were, within this documentary. It is sometimes difficult to 
distinguish when the characters are in the present narrative and when they 
are in the past. Thus, there is a strategy of illustrating them with present 
and past traces side by side so that the reader can distinguish this. In this 
sense, there is a frame for the very narrative of that Gaza episode within 
the “documentary” Footnotes in Gaza. 

In Footnotes in Gaza, other simple representations of media can become 
complex as the reader seeks to deepen the understanding not only of the 
narrated historical event, but also of the reasons why it has become a foot- 
note. Many television show illustrations, for example, add sense to the 
story, such as news about the war at the time when the comic story was 
produced, and an interview with US President George W. Bush (Fig. 4.5). 
In choosing to illustrate violent scenes that were aired on television, or 
that specific interview, there is an intention to produce significance related 
to the meaning of the story, as the narrator warns us: “Palestinian cameras 
do not hesitate to show the open wounds on the victims’ chest” (Sacco 
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Fig. 4.5 The television shows an interview with US President George W. Bush 
(Sacco, Footnotes in Gaza, 2009: 136). Copyright Joe Sacco. All rights reserved 
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2009:126). The reader needs to observe the illustrated scenes—not just 
the illustration of the television set. There is an iconic image of the zoom- 
in on the President that carries indexicality due to the easy recognition of 
him, while the television frame disappears, transforming a simple represen- 
tation into a complex one, and finally a transmediation of the television 
show onto the comics page. The names of newspapers and television chan- 
nels and programs as specific media products are also important, as they 
have different editorials and ideologies. Thus, they are no longer just rep- 
resentations of technical media; they make sense in the setting of Footnotes 
in Gaza itself. 

Biography comics also feature complex media representations, although 
they may not always be perceived if not supported by the perceiver’s rep- 
ertoire. Dotter of Her Father’s Eyes includes several, held by the core inten- 
tion of the work, which is to weave relationships with James Joyce’s style 
and to make direct references to his work. However, there are many allu- 
sions that the perceiver must notice and understand. 

The popular Irish folk song “Finnegan’s wake”, whose narrative 
inspired Joyce’s Finnegans Wake, tells the story of a drunkard named Tim 
Finnegan who falls off a ladder, breaks his head, and is resurrected at his 
funeral when friends pour whisky over his dead body. Composed in 1850 
by an unknown author, the story of the popular song is present in the first 
chapter of Joyce's book and also in Dotter of Her Father’s Eyes. In the pho- 
tographic montage that ends the comics, the chorus of the song— Wasn't 
it the truth, they told ye lots of fun at Finnegan's wake?"!*—is subtly 
placed on the last page of James Joyce's Finnegans Wake, which reads “My 
cold mad feary father" (Joyce 1992 [1939]: 628), the same sentence that 
triggers the story of Dotter of Her Fathers Eyes (Talbot and Talbot 2012: 
3). The representation of the Finnegans Wake media product, therefore, is 
significant not only in the simple direct quotation of the passage but also 
in the significance of a circularity, interweaving in the sequential structure 
of the comics the stories of Lucia, Joyce's daughter, and Mary, the pro- 
tagonist of the comics—and also the present and the past, in the connec- 
tion between the end of Lucia's story and the beginning of Mary's. 

In the introductory and final pages of Dotter of Her Father’s Eyes, we 
find photographs of the drafts by Atherton, the protagonist's father, dur- 
ing the creation of his book about Joyce, Books at the Wake (1959). The 
photos show changes in Atherton’s book summary and pen-written notes 
showing the creative process. Apparently, the notes are made on a printed 
copy of Finnegans Wake. Representing this process of creating the book, 
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Mary weaves it into her relationship with her own father, who lives apart 
from the family, inside his office, reading and paraphrasing Joyce. In this 
sense, this complex representation of the media product The Books at the 
Wake does not draw an intermedial reference related to Joyce’s work itself, 
but primarily to the protagonist’s own “structure” or “process of creation”. 

In To the Heart of the Storm, the sequential structure of the narrative 
simulates the flashback of cinema by illustrating the protagonist looking 
through a train window (Fig. 4.6). As he observes the landscape, memo- 
ries of the moments lived in those city locations are triggered: the scenes 
of the past are now “projected” through the graphic images of the illustra- 
tion. The train trip refers to the spatiotemporal mode of the story, in 
which the train window is the frame of this story canvas, as if a movie were 
played before the protagonist’s eyes. 

In Persepolis, the veil worn by women is an important graphic element 
to show the loss of identity, because it is very difficult to identify the 
women when wearing it. To stand out, Satrapi presents herself illustrated 
in one of the panels; while, in the other, her image is cut out, alluding to 
a strategy of photography in which a cut can be made at the time of pho- 
tographic recording or later in the process of development or in digital 
manipulation (Fig. 4.7). Thus, it seems that the “photographer” left her 
out of the picture with her friends. Another complex representation of 
media in Persepolis is in the illustration of the comics media product 
Dialectical Materialism, the protagonist’s favorite story. In depicting 
excerpts of Iran’s history, Satrapi reinvents it from her interpretation, 
weaving connections with her religious beliefs and thus creating her own 
dialectical materialism. 


43.3  Transmediations 


Transmediations are much less frequent in the comics analyzed. As we 
have already commented, this might be explained by the very characteris- 
tics of this kind of media transformation. First, transmediations depend on 
the possibilities of adapting the characteristics of one media to another. 
When the source and target media modality modes are the same, transme- 
diation involves less adaptation, while the opposite may even make trans- 
mediation impossible. Secondly, we understand that transmediations of 
parts of media products can often function as complex representations of 
media, as the producer seems to be willing to make that “piece” dilute 
“within” the media product. As for representations of traits—or 
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Fig. 4.6 The flashback of cinema by illustrating the protagonist looking through 
the train window (Eisner, To the Heart of the Storm, b: 7). Copyright Will 
Eisner. All rights reserved 
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THIS IS ME WHEN | WAS 40 
YEARS OLD. THIS WAS IN 4980. 


AND THIS IS A CLASS PHOTO. I'M SITTING ON THE FAR LEFT S0 
YOU DON'T SEE ME. FROM LEFT TO RIGHT: GOLNAZ, MAHSHID, 
NARINE, MINNA, 


Fig. 4.7 Satrapi presents herself, and her image is cut in the other illustration 
(Satrapi, Persepolis: The Story a Childhood, 2004: 3). Copyright Marjane Satrapi. 
All rights reserved 


characteristics—of qualified media types, these are (simple or complex) 
representations of media, as we have already shown here. As we have also 
commented, the boundaries between complex media representations and 
transmediations can be subtle. This is indicated by the fact that, in order 
to analyze the complex media representations and transmediations, it is 
necessary to interpret the media product in a more attentive and thorough 
way than merely perceiving the presence of other media. Triggering inter- 
textual repertoires and knowledge about how media work is also impor- 
tant for understanding complex media representations and transmediations, 
and for recognizing media types and media products being cited, alluded 
to, or transformed. 

By analyzing the photographic image of Satrapi’s mother in Persepolis 
“inside” a newspaper page (Fig. 4.2), for example, we easily identify a 
representation of newspaper media—as a type of printed media—and also 
the representation of a product of photographic media “printed” in the 
newspaper. However, in the previous frame, the same image appears with- 
out the frame of the photograph, because the frame is constituted by the 
typical comics panel. This means that the photograph was first transmedi- 
ated into the comics through the illustration, and then transmediated into 
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the newspaper that is represented in the comics. It is easier to understand 
this process if we accept the fact that this photograph exists outside the 
media product—in extradiegetic reality—and was actually published in a 
newspaper. 

In Dotter of Her Father's Eyes, Mary, the protagonist-narrator, adapts 
excerpts from the biography Lucia Joyce: To Dance in the Wake (Shloss 
2003) to her story. These adaptations are intramedial when she recounts 
parts of Lücia's biography through the verbal, symbolic elements of the 
comics; and they are intermedial when she recounts the biography through 
the iconic elements of the comics (cf. the distinction between media trans- 
lation and media transformation in Ellestróm 2020: 73-74). The case of 
photographic images is exemplary, as the illustrations often correspond to 
photographs inside or outside the book; they may be published on the 
internet, for instance. The scene illustrations of Lucia's dancing perfor- 
mances are transmediations of these photographs, but the reader must 
compare them, and scrutinize Lucia's biography and Mary's autobiogra- 
phy, to understand them as such. 

For both biographical and journalistic comics, it is vital that the docu- 
ment or photographs represented by the photographic images (better yet, 
copies of the photographs) are then transmediated to the illustration, 
making the indexicality surpass the typical iconicity of the illustrations 
(since the graphic images are iconic signs of objects). This is because biog- 
raphy and journalism comics need to demonstrate that illustrated images 
are not merely imagined; they can also represent something that really 
exists. Thus, they show the photograph and, later, the same image illus- 
trated. This is how, in Dotter of Her Fathers Eyes, a piece of the content of 
a photographic image—Atherton’s identity document—is transmediated 
for illustration (Fig. 4.8). 

Transmediations can also be noticed when the content of a certain 
book or newspaper is read—shown—by one of the characters. The illus- 
tration of Mary Talbot reading Carol Schloss's book on the train rein- 
forces the idea of a transmediation of the contents of the biography into 
the story being told in the comics. When Mary /istens to the interview 
with American writer Sylvia Plath, broadcasted on BBC in 1962, the 
excerpt from the poem that Plath recites live— Daddy, daddy, you bas- 
tard, Pm through" (Talbot and Talbot 2012: 36; Plath 1992: 222)—is 
transmediated again to written verbal media in comics. The 
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Fig. 4.8 Atherton's identity document transmediated by the illustration (Talbot 
and Talbot, Dotter of Her Father’s Eyes, 2012: 2-3). Copyright Mary M. Talbot 
and Bryan Talbot. All rights reserved 


transmediation of this exact passage from the poem is important because, 
written just before Sylvia Plath committed suicide in 1963, it speaks of a 
girl with Electra complex and her father, who died while she still thought 
he was God. 

In In the Heart of the Storm, the covers of some newspapers are quite 
noticeable in their details, showing—in addition to the names of the news- 
papers—titles, headlines, and even dates. It was not possible to research 
whether these covers are real or fictional, but it is not unlikely that they 
convey elements of newspaper covers that were indeed printed. Transmedia 
evaluation always requires that we carry out such comparisons between 
source and target media products. 

In Le photographe, the proof is used as a narrative resource, in which the 
frame of the photographs is kept, but transformed in the proper sequence 
of the comics.? The selection of the photos, made from the original proof 
by markings, indicating cuts in the photographs or the rejection of some 
of them, is maintained (Fig. 4.9). The inclusion of these rejected photo- 
graphs maintains the dynamics of the proof media type and characterizes 
their integral transmediation to the comics. One can see that the proof's 
frame disappears as it turns into comics on some pages that are composed 
entirely of photographs from this proof. The proof transmediation, which 
ends the comics, includes pen-written annotations and organization labels, 
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Fig. 4.9 The selection of the photos made from the original proof by markings 
(Guibert et al., Le photographe. Tome I, 2003: 62). Copyright Emmanuel Guibert, 
Didier Lefèvre and Frédéric Lemercier. All rights reserved 
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simulating the characteristic annotations of photographic evidence present 
in the proof. When photographs are isolated, taken from these cuts, we 
can still understand that they are transmediations as they become frames, 
when the photo frame becomes the frame of the comics. 

The same happens in I mondo de Aisha, when photographs of Yemeni 
women, whose testimonials compose the story, lose their frame as they are 
transmitted to the comics as photographic images, showing labels with 
their names on them. In another part of I/ mondo de Aisha, transmediation 
happens the other way around: the illustration of a scene in one of the 
comics is transmediated to a camera screen, becoming digital photography 
(Fig. 4.10). Of course, one may also understand that by placing the image 
that “shoots” the illustrated scene “inside” the camera screen, the graphic 
image simulates digital photography. 

As previously pointed out, complex representations and transmedia- 
tions often merge, and when parts of media products are transmediated, 
these transmediations are often part of media representations. Frequently, 
rather than transmediating a specific content of a media product, the 


PODEMOS TIRAR 
MAIS UMA? 


Fig. 4.10 The illustration of the same scene on a camera screen (Bertotti, J/ 
mondo di Aisha, 2008: 58). Copyright Ugo Bertotti. All rights reserved 
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objective is to represent media types or to show different media, under- 
standing that the media, by themselves, represent the world; they are indi- 
ces of reality. That is, the different mediations of the document (media 
representations) are highlighted, instead of its content (transmediations). 
In Footnotes in Gaza, excerpts from the UN document are sometimes 
transmitted to the illustration of printed pages, sometimes narrated by the 
journalist. At times, these graphic images of documents have titles in 
a different bold font that appear to be watermarks of the document but 
which, when transmitted to the illustration, are highlighted. 


4.4 | CONCLUSION 


We may have been redundant in some statements throughout this chapter, 
but case analyses are most effective when, from theoretical propositions, 
repetitions and frequencies (or infrequencies) can be shown. To end the 
article, we here summarize what we take to be vital results of our 
investigations: 


* Media representations are important for the construction of indexi- 
cality in the journalistic and biographical comics analyzed. 

e In journalism comics, simple representations of technical media and 
media types are frequent and central for the representation of jour- 
nalism, its agents, and its practices. 

e In biography comics, simple representations of photography media 
are important for building biographical quality, by adding indexicality. 

e In journalism and biography comics, the function of simple repre- 
sentations of real media types and media products is to place charac- 
ters and stories in real historical contexts. 

* [n journalism and biography comics, complex representations of spe- 
cific media products (such as TV shows, literary books, documents, 
and reports) often aim at bringing out elements of the extradiegetic 
universe that relate to fabular content; such complex representations 
of specific media products often make a lot of sense for the story 
as a whole. 

* [n the analyzed journalism and biography comics, complex represen- 
tations of media types, which evoke qualities and strategies of quali- 
fied media types, often merge into the structure ofthe media product 
that represents them. 
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In the analyzed journalism and biography comics, complex media 
representations make the meaning of the story multifaceted in its 
relationship with the media. 

The dilution of the qualities of one medium into another can be 
understood either as media representation or as transmediation. We 
believe that understanding it as transmediation requires a comparison of 
source and target media products and an analysis of adaptation. 
Representations of parts of media products can be understood as 
transmediation as long as they maintain the autonomy that enables 
them to be identified as such and understood in their significance. 


NOTES 


. We understand all nonverbal images of comics as graphic images, divided 
into graphic images of photography—simply called photographic images, 
to differentiate from photography—and illustration, related to “drawing”, 
which comprises the technique of producing handwritten images or imitat- 
ing the manuscript. It is important to point out that a graphic image of a 
photograph can also be a transmediation when the photo frame does not 
appear—as if the page were the photograph itself in an album. 

. From here on, we will treat comics as a qualified media type, according to 
Ellestróm's assumptions (2020). 

. Free translation from Portuguese into English. 

. Free translation from Portuguese into English. 

. At most, photographers were able to portray combatants resting and 
empty battlefields, as they were travelers *bent under the weight of over- 
sized equipment and forced to carry the laboratory—literally—with them” 
(Sousa 2000: 27). All excerpts by Sousa are translated from Portuguese 
into English by the authors of this article. 

. Joseph Pulitzer's Tbe New York World was the most popular American 
newspaper of the late nineteenth century, reaching a circulation of 600,000 
daily in 1896. Its main competitor was William Randolph Hearst's The 
New York Journal. The famous award given to writers and journalists in the 
USA was named after Joseph Pulitzer, while William Randolph Hearst’s 
biography inspired Orson Welles's 1941 film Citizen Kane. 

. The character Yellow Kid was the reason for the creation of the term “yel- 
low journalism", which referred to sensational journalism. The populariza- 
tion of comics, symbolized by the success of the yellow-nightshirt character 
with the general public, sparked “a reaction from conservatives who feared 
mass dissemination through an increasingly popular press, increasingly 
available to all" (Moya 1977: 36). 


o0 


10. 


11. 


12. 


13. 


14. 


15. 
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. In the USA, they are called funnies. 
. The graphic novel terminology was initially used by Eisner to generically 


designate a way of producing stories that would enable comics to demon- 
strate their creative capacity aligned with literature by using concepts of 
literary creation. Later, graphic novel came to designate the publication of 
comics with a different format (usually in book form with many pages) and 
a graphic quality compared to the best printed books. 

In the preface of Palestine, Sacco responds to accusations that he portrays 
only one side of the conflict. He states that “this is a correct judgment 
about the book, but it does not affect me. My position has been and still is 
that the view of the Israeli government is already well represented by the 
mainstream US media and is warmly defended by almost every elected 
politician in the United States" (Sacco 2009). 

As we do not have access to the isolated photographic news report that was 
published, we have no means of analyzing this adaptation. 

This clear visualization of the simulation, in which “photographs” fly 
through the window, was an exception among the complex representations 
of media types. In the rest of them, we can see a dilution of the character- 
istics of other media. 

Event described by the linguist, philosopher, and political activist Noam 
Chomsky (1999). 

From the original: *Isn't it the truth I told you? Lots of fun at Finnegan's 
Wake" (Talbot and Talbot 2012: 89). 

The photographic proof has many names, but it is basically a collection of 
small-size printed photos for a page, newspaper, magazine, book-filling 
project so the best photos can be evaluated and selected. Sometimes the 
negative itself can be a rough cut or copy. 
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Towards an Intermedial Ecocriticism 
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5.1 INTRODUCTION 


Let me begin with a question. What, if anything, unites the following 
three, very different, media products: a scientific article called “Trajectories 
of the earth system in the Anthropocene” by Will Steffen and his col- 
leagues (Steffen et al. 2018), which discusses aspects of what the authors 
call the Anthropocene epoch; Chinese director Zhao Liang's semi- 
documentary film, Behemoth (2016), which depicts the effects of coal and 
mineral extraction in China; and, finally, the collection of short stories I 
am with the Bears (Atwood et al. 2011)? Apart from being written, 
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produced, and published within a few years of each other, the obvious 
common denominator is that they all respond to and in different ways aim 
to represent different aspects of the current planetary ecological crisis as 
defined by, among others, atmospheric chemists, climate scientists, and 
geographers. They are part of the broad field of ecomedia that I will 
define later. 

To put it very briefly, the natural sciences work by meticulously collect- 
ing data, which are often based on measurements taken by sophisticated 
technical apparatuses, in and outside laboratory facilities. The data are, 
since the massive breakthrough of immensely strong computing powers, 
analysed by way of the now ubiquitous computer simulations (Roundtree 
2014). This enables scientists to produce descriptions of extremely com- 
plex conditions not only as they are at the present moment but also in 
relation to the past and the future and, in the case of ecologically related 
threats, the data are used to predict future scenarios. Most of such work 
and the discussions around it take place in the confined spaces of academic 
journals and scientific conferences, and this is how most non-specialists 
get to know about scientific results and research—via printed or digital or 
other media forms, often in relation to fairly ‘spectacular’ breakthroughs 
or particularly frightening predictions. 

In other words, for the natural scientific results to make any impact 
outside the relatively narrow circles of scientific publications and debate, 
the results need to be communicated, and communication per se involves 
media products with different possibilities and limits. Newspapers, popu- 
lar science journals, and documentaries are some of the well-known media 
forms that communicate information about scientific research, but poetry, 
science fiction novels, feature films, or art exhibitions can do this too. That 
the results of scientific research need to be communicated to the public is, 
of course, a known fact in several well-developed disciplines, including risk 
communication, health communication, science communication, and cli- 
mate change communication studies.! These fields are based on a tradition 
of communication studies which are particularly well suited to analyse the 
history and impact of journalistic media and mass medial forms, but that 
have not traditionally been interested in aesthetic communication. 

In this chapter I will, like these fields would normally try to, take as the 
crucial, rather commonsensical, starting point that it is indeed media 
products—three examples of which I mentioned to begin with—that 
communicate basic ideas about the ecological condition of the planet to 
most people. I agree with these fields that it is important to investigate 
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these on a sound theoretical and methodological basis, but, in my readings 
in the field, I have not yet come across a satisfying theory or method which 
describes and analyses (and thus compares) media products from the per- 
spective of the broad spectrum of the communication of science that goes 
all the way from, for instance, the United Nations’ Intergovernmental 
Panel on Climate Change (IPCC) reports to a science fiction feature film, 
a newspaper article, or a board game representing the ethical problems 
that are emerging from a warming world. Or the three examples men- 
tioned in my introduction above. 

It is my aim, in this chapter, to suggest a theory and method that are 
able to do exactly this: I call this position ‘intermedial ecocriticism’, and 
the most general idea of intermedial ecocriticism is the conviction that the 
ecological crisis is not a problem or a condition restricted to investigations 
in the natural sciences, or that possible solutions to the crisis can be 
reduced to technological solutions. The humanities need to play a role in 
the question. 


5.2 SKETCHING THE BACKGROUND 


The immediate problem of analysing and comparing the three examples 
mentioned above is that they clearly fall into three very different catego- 
ries. In the intermedial research tradition I belong to, the three examples 
are called “media products”, and they quite clearly belong to three differ- 
ent “qualified media types” (Ellestróm 2010, 2020). It is the fundamental 
differences in qualified media types that make it hazardous to make well- 
founded critical comparisons between them, unless the three different 
media are simply, and quite crudely, considered to be different boxes con- 
taining the same content. This is probably why the communication fields 
mentioned above have mostly refrained from making more substantial 
comparisons across media borders. Therefore, the question is which 
options are available if the ideas concerning the ecological crisis, as these 
are represented in a wide array of media forms, are to be analysed in a 
fruitful way. 

In this chapter, I argue that it is possible and even necessary to combine 
two academic disciplines, intermedial studies and ecocriticism, in order to 
face the challenge of analysing, discussing, and comparing texts like the 
three examples mentioned. Instead of using the communication studies 
approach, which has mostly been developed on social sciences 


120 J. BRUHN 


foundations, I have been looking for suitable theoretical foundations in 
what is often called the ‘environmental humanities’ or ‘green humanities’. 

I work from the hypothesis that, in general, the environmental humani- 
ties tradition is haunted by a methodological impasse, in that it often tends 
to do one of two things. The first is that in terms of media types, and thus 
academic disciplines, there is a tendency to ‘compartmentalize’ ecological 
questions, as if the ecological crisis respects the borderlines defined by 
media types or academic departmental traditions. The second is that when 
cultural theorists with broader interests that span different media types 
and academic disciplines examine ecological representations, they often do 
so, I will argue, without a sound methodological and theoretical ground: 
it takes a well-defined common ground to compare, say, a scientific article 
with a science fiction film. This is why, in what follows, I argue that the 
environmental humanities could be supplemented with an analytical tool 
that is sufficiently broad to analyse several media types and sufficiently 
fine-grained to do this in a detailed way. 

I do not wish to give the impression that there are currently no envi- 
ronmental humanities approaches that go in this direction, though. One 
such approach has been suggested by the cognitive cultural scholar Alexa 
Weik von Mossner. In an attempt to overcome some of the problems 
inherent in the field, she recently defined “environmental narratives” as 
“any type of narrative in any media that foregrounds ecological issues and 
human-nature relationships, often but not always with the openly stated 
intention of bringing about social change” (Weik von Mossner 2017: 3). 
This very broad definition (depending on how widely the notion of narra- 
tive is defined) may be compared with one of the shortest but, at the same 
time, most comprehensive definitions of ecocriticism, which has been pro- 
vided by the influential literary ecocritic Gregg Garrard. Ecocriticism, for 
him, is “the study of the relationship between the human and the non- 
human, throughout human history and entailing critical analysis of the 
term ‘human?’ itself? (Garrard 2012: 5). Garrard’s and von Mossner’s defi- 
nitions are valuable because they open a possible perspective of a broad 
environmental investigation across media, as opposed to the many produc- 
tive attempts to analyse environmental questions in one, single media 
type, not only in literature (the originating and still-dominating discipline 
in ecocriticism) but also in a number of other aesthetic media? However, 
even if these descriptions raise the hope of a truly cross-disciplinary eco- 
critical approach that can be adopted across, in principle, all imaginable 
media types, it has not really been fulfilled yet. In von Mossner’s study, 
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which is built upon a cognitive narratological foundation, the admirably 
broad corpus is restricted to fictive and non-fictive literary and cinematic 
narratives. And even if Garrard’s definition is potentially cross-medial, his 
own corpus is mostly literary material. 

What is needed, I argue, is an approach that can analyse and compare 
media products across, in principle, all media types and, for this purpose, 
the term ‘ecomedia’ has been proposed. Since this suggestion lies close to 
my own interests, I will discuss this position in some detail. According to 
Simon Estok, the term includes “any media that deals with environmental 
issues, implicitly or explicitly”. It is a reference to ‘Ecomedia studies’, 
defined by Stephen Rust as “a historically situated, ideologically moti- 
vated, and ethically informed approach to the intersections, of media, soci- 
ety, and the environment” (quoted in Estok 2017: 17). The notion of 
ecomedia studies is clearly affiliated with my idea of intermedial ecocriti- 
cism, but there are major differences. The field is described in the anthol- 
ogy Ecomedia (Rust et al. 2016), which defines itself as a textbook (it 
includes working questions and fact boxes). It offers a broad survey of 
qualified media types, including photography, cinema, advertising, new 
media, and gaming, as well as other, mostly mass medial forms, which are 
discussed under three headings. “Frames” deals with visual media, “Flow” 
deals with broadcast media, and “Convergence” deals with the mixing and 
transforming of mostly non-aesthetic ‘new’? media in contemporary 
culture. 

Ecomedia offers an admirable up-to-date discussion of crucial aspects of 
ecocritical discussions regarding a number of qualified media types across 
the globe. However, the book refrains from defining media, mediality, or 
mediation in any sustained form (it does not do so in the general introduc- 
tion to the book nor in the three opening chapters that define Frames, 
Flow, and Convergence). As a result, the book provides ten well-structured 
and very informative medium-specific descriptions of specific media forms? 
ways of relating to environmental questions through engaged case studies, 
but does not formulate general notions that enable comparisons and dis- 
cussions across the borders of different media types. So, what the book 
admirably gains in flexibility, global breadth, and illuminating examples, it 
lacks in systematicity and thus comparability. In what follows, ecomedia 
refers to the entire corpus of representations dealing with ecological issues, 
but it does not refer to any particular theory or analytical method. 
Regarding intermedial terms, I use the term ‘ecomedia’ to signify repre- 
sentations of environmental issues in any specific media product that in 
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any qualified media type engage in representing in fiction or non-fiction 
(or mixing these) either the reasons behind or the results of the environ- 
mental threats; Ecomedia representations include the notion of the effects 
of what has been called the Anthropocene epoch, which necessarily 
includes a questioning of the distinctions between human and non-human, 
and between culture and nature. 

In respectful but critical opposition to the Ecomedia position, I will 
suggest a cross-medial ecocritical approach, where media products from a 
wide range of media types could not only be analysed but also critically 
compared in meaningful ways. But what would such a method look like? 

My fundamental suggestion seems simple: I wish to investigate the 
broad corpus of ecomedia by applying intermedial terminology. That is, I 
aim to show that the rich and deep insights of contemporary ecocritical 
thinking can be combined productively with the analytical strengths of 
intermedial studies. As mentioned above, this is what I call intermedial 
ecocriticism. 


5.2.1 Three Basic Intermedial Points—And the Question 
of Representation 


I will refrain from setting out in any detail Ellestróm's notion of the four 
modalities of media or the distinctions between basic media types, quali- 
fied media types, and technical media of display, which are all clearly 
explained in Ellestróm (2020). Elsewhere, I have constructed an analytical 
model based on Ellestróm's ideas (Bruhn 2016; Bruhn and Gjelsvik 
2018). What I do want to stress is that Ellestróm's modelling of media 
makes it possible to gain a fundamental understanding of three basic inter- 
medial ideas that I consider to be part of the essential infrastructure of 
intermedial studies. The first idea is that all media products are medially 
and modally mixed, which of course is an idea that other researchers have 
been well aware of but have often described in more intuitive ways 
(Mitchell 1994; Bruhn 2010). It is tempting to jump to the conclusion 
that this would mean that there are no such things as media types (or 
genres, or other comparable categories), but this is not the case: almost all 
media products are, according to their definition, part of a qualified media 
type, but these types are not only defined by their constellation of media 
and modalities; they are also contextually and operationally defined. The 
second idea, therefore, involves talking about a contextualized medium 
specificity, meaning that despite the mixedness of media, all media 
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products, according to their definition, can be characterized with a weak, 
so to speak, or temporary medium specificity: any qualified media type 
offers temporary “affordances”, that is, limits and possibilities (Kress 
2010) that enable some things to be expressed and hinder other things 
being expressed. The third, general intermedial idea is that, apart from 
having a descriptive analysis and contextualized definition of the basic 
media aspects, it is also useful to work with a key analytical distinction 
between media integration and media transformation.? 

This distinction is a result of the a prioric idea of intermedial studies, 
namely that all media are medially and modally mixed, which naturally 
leads to the question of how the a prioric mixedness of all media texts may 
be analytically approached. It is not particularly interesting to simply dem- 
onstrate the mixedness of media products; instead, the mixedness needs to 
be described and analysed. From an analytical, pragmatic point of view, 
the most important dimensions are media integration and media transfor- 
mation—the transformation dimension is particularly important in inter- 
medial ecocriticism. 

Broadly but briefly speaking, the media integration perspective applies 
to phenomena where two or more medial forms or modes coexist in the 
same media product at the same time. Examples could be if a painting by 
Cézanne is represented in a film and jazz is playing in the background or 
when images and written words coexist as parts of a scientific article. On 
the other hand, the mediam transformation perspective investigates how 
medial content or form is transformed from one medium to another in a 
temporal process. For instance, when a scientific idea (like oxygen) is 
drawn as connected atoms on a blackboard in a classroom, when a novel 
by Jane Austen is turned into a film, or when the idea of climate change, 
as expressed in scientific articles, is represented as the main theme in a 
contemporary environmental documentary or a feature film. 

Media transformation, according to its definition, contains a temporal 
perspective. First there is a novel, then it is turned into a film; first there is 
film, then it is turned into an amusement park; first there is a painting, 
then there is a poem representing this painting, and so forth. In this large 
corpus of cultural objects and processes, introduced and discussed in 
Linda Hutcheon's A Theory of Adaptation (Hutcheon 2006; see also 
Bruhn et al. 2013), the medial mix lies, so to speak, in the process: certain 
aspects of a novel (typically, themes, parts of the plot, certain characters, 
setting, etc.) are transported into a film, but certain aspects of the adapted 
work are necessarily left out or changed beyond recognition. The media 
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transformation process is transferring certain aspects while also transform- 
ing everything into a new media product (and a different technical and 
qualified medium). Analysing media transformation basically means 
understanding the interplay between medium specificity (what character- 
izes specific media) and transmedial aspects (that which crosses media 
borders). 

From an analytical point of view, it is helpful to divide all the inherent 
mixedness of all media products into either ‘temporal transformation’ or 
‘synchronous integration’, but this distinction is pragmatic rather than 
essential. Given the idea that all medialities are mixed, it follows, on the 
one hand, that all media products are in fact an integration of mediality or 
modality aspects. Given the fundamental idea of intertextuality, meaning 
that all texts are versions of earlier texts (Allen 2011), on the other hand, 
we may conclude that all medialities are, basically, the result of a 
transformation. 

Consequently, when performing an intermedial analysis on a specific 
media product or on the relations between media products (or when 
thinking about the general aspects of a media type), one might investigate 
either mixtures (integration) or traces (transformation). Thus, from a 
medial perspective, any media product, for instance, a literary text, is com- 
parable to the famous duck-rabbit illusions: depending on your analytical 
interest and investment, you can choose to perceive a media product either 
as a combination or as a process of transformation; both dimensions are 
inherent aspects of the specific media product. To get the fullest possible 
description and interpretation, one might combine the two approaches, 
but many specific analyses will typically focus on one of the two dimensions. 

At this point in my demonstration of the general analytical notions, I 
need to touch upon the question of representation, more particularly the 
representation of environmental issues, which has been a central concern 
in contemporary environmental humanities. Rob Nixon is an important 
commentator on this question, and he stresses the “formidable represen- 
tational obstacles” of describing global warming (Nixon 2011: 2). He also 
underlines that the effects of the ecological crisis, which he usefully and 
famously terms “slow violence”, is “a violence that is neither spectacular 
nor instantaneous, but rather incremental and accretive, its calamitous 
repercussions playing out across a range of temporal scales” (Nixon 2011: 
2). This means, basically, that the dramatic effect of the ecological crisis in 
general, and global warming specifically, is a phenomenon that is imbued 
with a set of complicated representational issues. In recent ecocriticism 
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concerning the representations of global warming, Timothy Morton’s 
idea of the ‘hyperobject’ (Morton 2013) has been perhaps too influential 
in convincing critics of the near-impossible task of understanding and 
therefore representing climate change as well as other aspects of the eco- 
logical crisis.* I intend to follow Nixon in acknowledging that the environ- 
mental crisis presents huge problems of both grasping and representing 
the crisis, but I do not find it useful to consider, for instance, global warm- 
ing to be an unrepresentable hyperobject. 

I prefer a more pragmatic understanding of the abilities of representa- 
tion. Representation, as I see it, is a fundamental part of human commu- 
nication: it is a process that uses media products to stand in for all sorts of 
phenomena of material or mental character, including so-called fictive and 
non-fictive phenomena. According to their definition, representations 
cannot be completely identical to what they represent, and representation 
is not devoid of performative aspects, so any representations necessarily 
function on a differentiating scale from higher to lower precision and 
effect, depending not only on the media products but also on the context 
in which the media products are produced and perceived. Language, visual 
communication, and any other semiotic forms of representation are often 
very efficient but not impeccable tools of communication. In principle, I 
do not wish to exaggerate the fundamental differences in how easy, or dif- 
ficult, it is to represent the coffee cup on the table beside me as I write 
these lines as compared to the effects of climate change or the victims of 
the so-called sixth mass extinction (for a broad discussion of the question 
of representation, see Hall et al. 2013). 

Let me try to sum up the first, general part of my exposition. Intermedial 
ecocriticism is intended to mark the outline of a broad-spectrum research 
goal. The most general background of intermedial ecocriticism is the con- 
viction that the ecological crisis is not a topic restricted to investigations in 
the natural sciences, or that solutions to the crisis can be reduced to any 
quick tech fix. Instead, as environmental humanities stresses time and 
again, the ecological crisis relates to key questions that the humanities ask 
and, consequently, ecological issues are "being reimagined as an ethical, 
societal, and cultural problem? (Yusoff and Gabrys 2011: 517). Or, to use 
a slightly longer quote from Ursula Heise's 2017 introduction to a large 
handbook on environmental humanities: 


The environmental humanities, by contrast [to the technico-scientific 
approach], envision ecological crises fundamentally as questions of socio- 
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economic inequality, cultural difference, and divergent histories, values, and 
ethical frameworks. Scientific understanding and technological 
problem-solving, essential though they are, themselves are shaped by such 
frameworks and stand to gain by situating themselves in this historical and 
sociocultural landscape. (Heise et al. 2017: 3) 


Intermedial ecocriticism, more specifically, contributes to the explora- 
tions in ecocriticism and environmental humanities by stressing that eco- 
logical questions, according to their definition, relate to questions of 
communication and thus representation, and by stressing in particular that 
environmental humanities and ecocriticism need a cross-disciplinary and 
cross-media analytic approach that matches the necessarily broad nature of 
the environmental crisis. 

By using the notion of intermedial ecocriticism, I aim to develop a 
method to analyse and interpret specific media products, or, on well- 
defined methodological grounds, to compare more than one media prod- 
uct across media borders. 

In other words, this is the broad outline of some of the outcomes that 
an intermedial approach to ecocritical questions may achieve, but it is 
important to stress that combining the field of intermedial studies with 
ecocriticism (or other fields inside environmental humanities) may lead to 
several rather divergent approaches. How ecocriticism and intermedial 
studies are combined depends, for instance, on the specific scientific or 
even ideological interests of the researcher: it depends on which audiences 
are meant to receive the results and whether the research idea is meant to 
be used in didactic contexts. In what follows, I sketch out one among 
many possible operational approaches of intermedial ecocriticism. 


5.2.2 Intermedial Ecocriticism: A Methodological Suggestion 


The obvious initial move, before any analysis is even begun, is to choose 
one or more media products with subject matters that relate to ecological 
issues. In short, the media products should be part of the corpus of eco- 
media. This must be followed by choosing a specific question or problem 
to investigate in the chosen media product(s). Topics to investigate can be 
anything from the representation of the ecological ‘crisis’ to narrative 
aspects of the ecomedia product, to the ways in which the media product 
facilitates human agency in the Anthropocene, to temporal patterns in 
Anthropocene representations. Or, and here I am referring to existing 
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ecocritical work, one can discuss whether and how the existential and psy- 
chological notion of ‘pretrauma’ exists in cinema and literature (following 
Anne E. Kaplan 2015), or different affective patterns in narrative as well as 
non-narrative media products (following Weik von Mossner 2017). It will 
come as no surprise that the initial choice of topic has important conse- 
quences, because it will direct the analysis of the media product (and if one 
works with more than one media product, the chosen topic will be the one 
that is being compared). To explain the approach, in what follows, I will 
use an imagined ecomedia product: a short, printed newspaper article 
which describes the melting glaciers in the Arctic, and the topic I want to 
explore will be examined by asking this question: how, in intermedial 
terms, are the melting glaciers represented in this article? 

After the initial choice of media product(s), as well as one or more top- 
ics to be analysed and perhaps compared, three consecutive steps follow: 
in the first, descriptive step, the main aim is to define and analytically 
describe the fundamental features of the media type in general and the 
media product in particular. In the second step, after having established 
the basic features, the aim is to specify, by way of an analysis and interpre- 
tation, the exact ways in which the affordances of the media product 
enable or disable specific aspects that are to be represented. In the third 
and final step, the results of the analysis can be compared to the analysis of 
one or more media products. 

In the analysis, I apply the descriptive media tools suggested by 
Ellestróm (2020), but let me again emphasize that the general features of 
the approach I have sketched out can be followed by using other analytical 
tools that facilitate analysis of texts and comparative investigation. There is 
a more detailed description below of the three-step working processes. 


Step 1 

Step 1 consists of, first, carrying out a descriptive analysis of the (often 
obvious) technical media of display that characterizes the chosen media 
product; here the researcher can choose the media of display through 
which he or she has received the media product, or the researcher can 
describe the typical or most widespread display function for the specific 
media product. With regard to my example, in the short article on the 
melting Arctic ice caps, the technical media of display is a paper version of 
a newspaper (or the internet version of this article, which would make the 
technical media of display the computer screen or the mobile phone 
screen). I will refer to the paper version here. 


128 J. BRUHN 


After this, what Ellestróm defines as the material, spatiotemporal, sen- 
sorial, and semiotic categories of the basic media dimension of the media 
product are described. The basic media aspects of the newspaper article 
that I take as an example would be that the material aspect would include 
paper and ink: the spatiotemporal aspect would include the fact that the 
conventional way to read the article would be to read the text in the 
sequence produced by the journalist, even though it is possible, with a 
written text, to read it in your own preferred way. In sensorial terms, a 
printed article is predominantly perceived through the eyes, and the fun- 
damental semiotic aspect of the newspaper article would include the fact 
that the verbal written text is mainly interpreted by way of its symbolic 
sign qualities; if there are photographs in the article, these would relate 
more to an iconic sign function (both aspects are interpreted according to 
Peircean semiotics). 

The aim of the first step is to establish a precise and detailed description 
which will enable the next step’s more interpretative actions. The descrip- 
tion is meant to result in a clear understanding of the different medial 
aspects of a text, which therefore makes the following interpretive moves 
easier—and, if necessary, it makes the comparison between media prod- 
ucts possible. 


Step 2 

Having established the qualified media type, the aim of step 2 is to analyse 
and interpret the media product using intermedial analytical tools. The 
media integration and/or the media transformation perspective can be 
applied, as well as other intermedial analytical and descriptive tools. The 
aim of the analysis is to specify the affordance of the qualified media type 
as this plays out in the specific ecomedia product regarding its ability to 
represent ecological topics. Generally, the media transformation aspect 
will loom large in many intermedial ecocritical analyses because all ecome- 
dia products are fundamentally, from an intermedial point of view, media 
transformations of scientific material into ecomedia. 

So, regarding the short, printed newspaper article mentioned above, 
from a media transformation angle it is an example of how a scientific 
measurement of ice caps, which has first been published in a natural scien- 
tific article, has been transferred and transformed by a journalist into the 
format of a news item. The interplay between the photograph and the 
written text of the article can be analysed from the media integration 
perspective. 
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The aim of this analysis is to understand which aspects of, in this case, 
global warming, as this affects the Arctic ice caps, a short newspaper article 
generally, and the chosen article specifically, can represent. To a certain 
extent, the analysis may confirm rather commonsensical and general con- 
clusions, but the advantage of the analysis is that it can be much more 
precise in pinpointing the exact affordances of the general media type and 
the specific media product. In my more extended examples below, I will 
demonstrate the usefulness of this step. 


Step 3 

Having established the specific affordances of the given media product, 
step 3 allows the possibility of comparing it with other media products 
with a directly comparable subject matter. If this step is taken, then the 
second, or third, or fourth media product must of course be subjected to 
the two steps described above before conducting the third, compara- 
tive step. 


5.3 COMPARATIVE CASE STUDY 


5.8.1 A CarbonBrief Article 


Having explained the mechanics of my method by way of an imagined 
example, I now wish to do a comparative case study of two actual existing 
ecomedia products. 

The specific topics I wish to pursue are as follows: 


(a) How, from an intermedial point of view, do the media products 
represent scientific research related to global warming? 

(b) How, from an intermedial point of view, do the media products 
employ strategies that make hard science directly related to the 
everyday experience of non-experts in climate science? 


The first ecomedia product I wish to analyse was published in August 
2019 on the website of CarbonBrief, which is an internet-based, fact- 
oriented media outlet situated in London. CarbonBrief publishes articles, 
comments, and other material related to global warming issues on the 
internet but not on paper: there is often a focus on aspects related to the 
UK, but the work has a clear global outlook. Material published on the 
site is based on new events relating to global warming, either events in the 
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world or scientific reports or new findings. The article I will analyse was 
written by Daisy Dunne and is called “Climate change made Europe’s 
2019 record heatwave up to ‘100 times more likely’” and was published 
on the website on 2 August 2019 (Dunne 2019). 

The general aim of this popular scientific article, which consists of about 
1500 words, one photograph, and two modified maps of Europe, is to 
represent to its readers the scientific hypothesis that the global tempera- 
ture on the planet is rising due to anthropogenic influence. More specifi- 
cally, the article argues that different dramatic results of this new planetary 
condition can be analysed in a new and more exact way by means of so- 
called attribution science. Attribution science has advantages compared to 
more traditional climate science, which has been very cautious regarding 
linking actual weather phenomena with climate change. The aim of attri- 
bution science is to link much more directly the concrete and often dra- 
matic weather phenomena with more general developments in the climate, 
typically by saying that a certain dramatic weather phenomenon has been 
made x times more plausible due to the larger effects of climate change. 

The question is, how does the article do this—and can it be meaning- 
fully compared to other media products with approximately the same aim? 


Step 1 

Given the fact that CarbonBrief is a web-only publication, any reading of 
the text will be done via a screen, either on a computer (that is how I read 
it) or on a mobile phone. In my reading of the text, the technical medium 
of display is therefore a laptop computer screen. 

The material aspect of the article, as displayed on a computer screen, is 
the two-dimensional space upon which one can read text and see images 
by way of the technical apparatus making things visible on a screen, for 
instance, by scrolling on the page. An internet connection is also necessary 
to be able to reach the website. Several colour photos and/or colour data 
visualizations are a natural part of all the articles on the CarbonBrief web- 
site, and these also form part of the material aspect. The cost of reproduc- 
ing images is much lower for online publications than for printed articles, 
so there tend to be more in the former. 

In this case, there is a large colour photograph documenting a man 
shielding himself with a newspaper during the heatwave in London in July 
2019 and two visualizations of historical heat patterns in Europe. 

The spatiotemporal organization of the media product is a bit compli- 
cated. On a superficial level, the article has many similarities with a printed 
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article. However, there are several highlighted words (twenty in all) in the 
article, which function almost like footnotes in a printed text, meaning 
that when the reader follows such a hypertext link, he or she is redirected 
to another text. However, the function of a hypertext reference, as it is 
used here, and a footnote is not identical. Whereas the function of a foot- 
note is, conventionally, to offer either a short additional remark and/or a 
specific reference, the hypertext link in a digital text instead redirects the 
reader to another article, which the reader gains access to by clicking. The 
reader can hit the back button on the browser to return to the “Climate 
change made Europe’s 2019 record heatwave up to ‘100 times more 
likely?" article, but he or she may get lost or simply become more inter- 
ested in the new website, so it is possible that there is no return to the 
original article. Or, to speak in spatiotemporal terms, the hypertext affords 
a temporal and spatial sequence which differs from the rather strict and 
relatively simple spatiotemporal sequence of a printed article. The rather 
strict sequence of reading the conventional written article is broken up 
into a more fluid and potentially less restricted (in space and time) form. 
Compared to the limited length of a written article (ending when the 
article ends), the web article is potentially non-finite, at least if one follows 
the links to new links and more new links. It is worth mentioning that 
most of the hypertext references are ‘in-house’, so to speak, to CarbonBrief 
articles that go into more depth about some of the topics in the article. 

The sensorial dimension of the digital article on the net does not differ 
significantly from that of a written text: the visual interface is perceived via 
the eyes. However, when clicking around in the article, bodily movement 
of the fingers on the keys or the mouse pad or on an external mouse is 
needed, which initiates a sensorial element of touch, too. 

The semiotic aspect of the article combines in intricate patterns the 
three sign relations suggested by Peirce (as do most media products). The 
written text must be read as symbolic signs designating verbal language, 
whereas the photograph and the two visualizations must be read as two 
kinds of mainly, but not solely, iconic representations. The initial photo- 
graph iconically signifies extreme heat, represented by a male figure, 
whereas the two following visualizations consist of maps of Europe upon 
which coloured areas symbolically signify (by symbolic colour codes) the 
heat at a given moment in the summer of 2019 and, in the next map, the 
historic development of heatwaves from 1950 to 2018 as compared to the 
summer of 2019. Major aspects of the media product rely on the indexical 
sign function in the sense that many of the most important facts of the text 
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as well as the visualizations are representations of indexical measurements, 
that is, natural scientific investigations with a variety of measuring instru- 
ments that are then collected and represented in the text. 

The first, descriptive, part of the analysis can now be summed up. The 
general media type is the popular scientific journal article, which, more 
specifically, is technically displayed by way of computer screens or other 
electronic devices. This technical aspect leads directly into the material 
aspect relating to the very basic possibilities that an electronic text offers 
(in comparison with a printed text). Important here is the fact that, for 
economic reasons, a higher number of photographs and other illustrations 
can be included. Also, the hypertext possibilities, understood as part of the 
spatiotemporal set-up, are crucial when it comes to referring directly, via 
links, to other texts. In sensorial terms, human sight dominates, but touch 
is implied, too. 

With this initial description of the media product, I hope to have exem- 
plified that, even though such a description is rather banal, it has the 
potential to clarify one thing: that understanding the basic medial set-up 
of a media product conveys significant aspects of it, even before going into 
the elements of the text that the cultural sciences would normally feel 
more comfortable thinking about, namely the content and form aspects: 
what it is about and how is it formed. 


Step 2 

Based on the descriptions created during step 1, Step 2 is meant to involve 
a discussion of the general affordances of the media type, and in particular 
the specific media product, to prepare for a specific discussion of the gen- 
eral ecological question and the specific topic. 

The popular scientific article, as a media type, is a classic example of 
science communication, and it partakes in what in science communication 
studies is understood as the “public understanding of science” (Burns 
et al. 2003). More specifically, the article is part of the field of environ- 
mental communication, which has exploded in recent years according to a 
literature review (Comfort and Park 2018). 

The purpose of the article is to represent complex scientific research 
about findings in climate science in general, and about so-called attribu- 
tion science in particular, to the general public. The article tries to cater 
for non-specialist readers and add to the public understanding of science, 
as mentioned above. This can be reformulated in intermedial terms; the 
writer of the CarbonBrief article, Daisy Dunne, conducts a media 
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transformation by moving content or ideas from the medium of scientific 
article(s), which is the originating background or source, into the new 
media type, the popular science article. A basic rule of popular science 
communication of complex research results is to simplify them to make 
them understandable. Therefore, there is an immediate element in the 
media transformation that consists in picking out and ‘translating’ techni- 
cal terms, highly formalized terminology, and numbers that are difficult to 
grasp into something else. 

Simultaneously with simplifying and picking out material, the writer 
faces a more constructive task, namely to find appropriate ways to repre- 
sent the ideas which have been researched in the natural sciences. In inter- 
medial terms, this means that the media transformation aspect connects to 
the media integration aspect. The question of representing the facts of the 
scientific findings can be partly solved by choosing and combining differ- 
ent media aspects and modalities. 

The most important combination aspect in the article is between—in 
Peircean terms—the signs that are mainly meant to be interpreted as iconic 
signs (the photograph and the two coloured maps) and the written text 
consisting of mainly symbolic signs. The written text dominates in the 
meaning making of the message (in the sense that the text could more or 
less stand alone and still be meaningful without the images—the opposite 
is not the case), but the iconic features are very important. The title of the 
article, written on top of the photograph, designates the entire article’s 
content: climate change made Europe’s 2019 record heatwave up to “100 
times more likely”. But it is also a complex verbal utterance: taking as its 
starting point the current heatwave, it connects the dramatic weather 
event to a larger historical trend and stresses the most important conclu- 
sion of the underlying research: that heatwaves are becoming more fre- 
quent and warmer as a result of global warming. In addition, the title 
contains a quote which has no author, although it will become clear where 
it comes from if one reads the article. The rest of the article substantiates 
this headline and supports the claim with the aid of three main forms of 
support: (a) quotes and reported quotes from researchers; (b) the material 
referred to in the hyperlinks, of which several are current measurements of 
the temperatures around Europe; (c) visual material. The quotes and the 
material in the hyperlinks build up an argument using theoretical and his- 
torical claims combined with scientific measurements that together aim at 
producing the overall message, which has a narrative form: it starts from 
the individual experience (heatwave in London) and leads to a more 
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general fact (heat all over Europe), which are both part of a more elabo- 
rate trend (heat in Europe over time). This is an argument that clearly 
goes from the specific to the general. 

The iconic material, on the other hand, is effectively organized to sup- 
port that narrative form. It starts from the individual, human-interest per- 
spective in the first photograph and leads to the second map of Europe 
where the individual aspect of the first image is, so to speak, made a condi- 
tion for many Europeans at the same time. Finally, with the second, his- 
torically oriented map, the current heatwave is seen as corresponding to a 
much more comprehensive pattern. The pictures tell a rudimentary story 
that supports, or, rather, doubles, the written text: the pictures ‘say’ that 
our individual experience must be taken seriously (photograph of man in 
heatwave), but that the heatwave was probably more than a mere coinci- 
dence—it was a European phenomenon (first map). Not only that, but the 
extreme weather can be explained by referring to historical climate models 
(second map), which give the initial individual experience a totally new 
meaning. 

The use of hyperlinks in the text is another combination aspect that I 
have already mentioned but which should now be understood in more 
depth. It should come as no surprise that the technical and material aspect 
of a text has effects on its meaning production, but in this case in particu- 
lar the spatiotemporal nature of the digital text has certain effects, which 
are best understood by comparing it to a conventional printed news arti- 
cle. Generally, and on the positive side, the journalist producing a digitally 
disseminated text can use coloured illustrations and photographs to 
accompany the text without incurring large additional costs and without 
technical problems or limits because of space, as compared to a printed 
article in a journal. From the point of view of the reader, this use of visual 
material may make the reading easier and more pleasurable. Apart from 
the overall graphic layout of the article, made possible by the digital plat- 
form, a digital text like this facilitates an easily navigated reference system 
to other digital texts, including references to other media (the BBC, for 
instance) as well as meteorological sites and scientific texts. Also, in order 
to avoid explaining all the scientific background facts, the CarbonBrief 
article includes several ‘internal’ references, that is, references to earlier 
articles by CarbonBrief. Finally, the article ends by referring to related 
articles where the points being made can be deepened or discussed. 

Making references to other news outlets or to already published mate- 
rial is not unique to the affordances that the specific technical media aspect 
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of digital journalism allows: the point is that reaching these references only 
takes one click. 

The final feature that distinguishes this article from a printed text is the 
fact that readers can comment on the text immediately; CarbonBrief 
employs a feedback system called Disqus, which prevents anonymous 
comments and only allows so-called comments from registered users. In 
the case of this particular article, at the time of writing (7 September 2019, 
a month after the publication of the text), only two comments accompa- 
nied the text: both were critical of the content of the article. Again, several 
of these features exist in a printed article in a newspaper or a journal or 
magazine. The difference is that the illustrations in the online article are 
cheaper to reproduce, the references to very comprehensive material out- 
side the text can be much more elaborate and easier to get to, and the 
comments from readers are directly included alongside the article in its 
present state instead of being, in a printed newspaper, part of a later edi- 
tion one or several days later, where the readers’ comments stand alone 
without the originating debated text. 

There are also limitations to this media type compared to a text in a 
printed newspaper. First of all, this article is relatively brief in terms of the 
number of words it contains, which may be part of the strategy of 
CarbonBrief to make their climate news both understandable and digest- 
ible, and although the length is limited, the breadth of material that the 
hyperlinks connect to makes the article almost infinite in size. Second, a 
downside to using hyperlinks is that the writer risks ‘losing’ the reader as 
he or she is making their way through the article if the reader follows one 
of the links and does not return to the originating article. 

After this long descriptive analysis of the intermedial aspects of the arti- 
cle, it is now possible to offer a more interpretative answer to the ques- 
tions I posed initially: 


(a) How, from an intermedial point of view, do the media products 
represent scientific research related to global warming? 

(b) How, from an intermedial point of view, do the media products 
employ strategies that make hard science directly related to the 
everyday experience of non-experts in climate science? 


In response to (a), when it comes to the representation of scientific 
research, I have already pointed to the obvious fact that the CarbonBrief 
article simplifies the research to a level at which it can be understood by 
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non-specialists. This means that complicated terminologies, comprehen- 
sive methodological discussions, and large datasets (or thorough descrip- 
tions of these) are not included in the article. The brevity of the article (as 
compared to most standard scientific articles) is a sign of this. The use of 
hypertext links retains the brevity of the article while offering more in- 
depth material to readers who are interested, for personal or professional 
reasons, in having both the ideas and the references as further informa- 
tion. The article manages, in the short space it occupies, to refer to the 
fundamental method of ‘attribution science’ as compared to climate sci- 
ence’s normal approach, and this gives the reader a feeling that the sci- 
ences are not static but are evolving, indeed progressing. The graphic style 
of the article (which is a typical intermedial phenomenon in itself, since it 
considers the visual set-up of the page, the layout, etc.) gives the article an 
appealing look that makes the reader feel invited to take part in the 
content. 

A large part of the appeal of the article has to do with the use of images: 
the first photograph makes the article directly link to related experiences 
that many Europeans (and others) have had both in 2019 and in 2018. 
And, as mentioned already, the three image sections of the article not only 
illustrate the article but also ‘tell’ a little story themselves while, at the 
same time, they give the article ‘space’ and ‘air’, which is part of its invit- 
ing style. 

In response to (b), the images are part of the article’s attempt to not 
only represent a ‘readable’ version of complicated scientific research but 
also to make the research relevant for non-specialists. Here, the initial 
photograph, in particular, is a hook; for many readers, experiences of 
extreme heat are familiar, and instead of focusing on the anecdotal aspects 
or the personal feelings of isolation that the heat causes (which is the angle 
of many mainstream reports on heatwaves), this article uses a personal 
experience first, but then makes it a much more widespread feeling, first 
on a European level (i.e., in space) and later on the level of a long histori- 
cal period (i.e., in time). 


5.3.2 Charlotte Weitze’s Novel Den afskyelige 
(The Abominable) 


Because of spatial restraints, I will offer a shorter version of the descriptive 
analysis of the second media product than of the first one, but this will still 
make a comparison possible. 
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First, I need to present the media product. Charlotte Weitze’s 320- 
page novel Den afskyelige (The Abominable) (Weitze 2019) is narrated by 
Heidi, a young nurse living in northern Scandinavia, probably Norway, in 
a relatively near future dominated by global warming. She tells the dra- 
matic story of how she falls in love with a man, Kenneth, who is apparently 
suffering from hyperhidrosis, which is abnormal sweat production, and 
she describes in detail how the couple struggles, with mixed results, to live 
a CO,-neutral life in a globally warmed and warming world. Later in the 
novel, the reader understands, and so does Heidi, that her lover Kenneth 
has no conventional diagnosis: instead his many symptoms stem from the 
fact that he is actually a descendant of a yeti. She leaves her lover, despite 
being pregnant, but later loses the foetus. After a dam collapses towards 
the end of the book, she hides on a mountain while trying to save her best 
friend, Mette, and her baby. However, Mette drowns in the rising waters 
and the book ends with Heidi taking care of the little baby, not knowing 
whether the mountain top on which they live is the only place on earth 
where it is still possible to survive. 

It is clearly a novel that deals with global warming, but in an entirely 
different register from that of the article previously discussed. It will hope- 
fully be possible to define the differences more precisely by way of an 
intermedial analysis. 


Step 1 

Step 1 of this shortened analysis will just briefly establish the basic medial 
facts about the novel. Regarding the technical media of display dimension, 
I read the work in a hardback printed book version; the book has a dust 
cover with an illustration that is identical to the one on the hard cover 
(designed by Alette Bertelsen), which combines verbal language with the 
dark profile of a looming animal figure while also alluding to a mountain 
with a tiny house on top, thus referring to the end of the novel. 

The basic material dimension of the novel is printed letters on paper 
pages, and the spatiotemporal set-up is that of most of narrative written 
literature (see Bruhn (2016): a forward sequence is suggested even though 
one can indeed read a novel from the end and towards the beginning or 
read it at random places. Besides turning the pages and holding the book, 
which requires the sense of touch, the most important sensorial aspect 
when reading a novel is of course using our eyesight to read. Apart from 
the dust cover, the dominating semiotic nature of the novel is the sym- 
bolic, linguistic sign function. 
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There are, to use Ellestróm's terminology (2020), qualifying and con- 
textual dimensions which place this book solidly in the tradition of writ- 
ten, narrative literature: the generic form of the novel (roman in Danish) 
is stated on the cover and on the title page. However, this is very unusual 
for a novel, at the end of the book, the reader finds a page of acknowledge- 
ments, thanking funding agencies and colleagues. In addition, Weitze 
extends her thanks to a number of experts in many different fields includ- 
ing disaster research, psychiatry, and studies of fanaticism and religious 
beliefs. These experts contributed to her 'research for an open micro- 
phone' for a radio show on Danish broadcast Radio 24syv. 

To sum up the first step: it is not difficult or surprising to categorize 
Den afskyelige as a written narrative novel; the question is how to analyse 
it from an intermedial perspective. 


Step 2 

How then, in the second step, can the text be analysed from an interme- 
dial point of view, including the fundamental axes of media transformation 
and media integration? Let me discuss the media transformation first. 

Because it is an ecomedia product, the entire novel can be said to be a 
media transformation of already existing ideas concerning the ecological 
crisis and global warming, as this has been put forward in a huge network 
of scientific, political, and popular scientific material. 

I mentioned that the novel has an (for literary fiction) unusual list of 
acknowledgements that includes not only friends, colleagues, and funding 
agencies but also a number of experts in several fields. Weitze thus explic- 
itly acknowledges that several aspects of the novel must be seen as media 
transformations from the fields of climate science, engineering (concern- 
ing dams), and fanatical (religious) psychological states, among others. 
From this point of view, it would be possible to analyse a number of spe- 
cific scenes in the book, as well as the overall theme of the book as the 
intermedial interplay between medium specificity and transmedial subjects 
(which was a way of defining the study of media transformation, as men- 
tioned above). In other words, it would be possible to identify which 
aspects of the forms and contents of the originating texts that may be 
transformed (the transmedial aspects) and which aspects that are only 
transported with greater difficulty (because of medium-specific con- 
straints). To exemplify briefly: the detailed research results of a scientific 
article cannot be referred to at length in the novel, but the two protago- 
nists can discuss why it is that a vegetarian diet is much more 
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climate-neutral than consuming meat (Weitze 2019: 162-163). Therefore, 
almost everything in this novel, from the overall thematic content to many 
detailed discussions and particular scenes of the novel, might be analysed 
as instances of media transformations from a number of scientific disci- 
plines into the supposedly fictive set-up of the novel. 

Regarding the media integration analysis, there are different ways to 
pursue this. For instance, it would be interesting to analyse the ways in 
which the novel represents media products from the mass media, how 
mass media products stand up against other medial representations, and 
how this particular relation influences the protagonists. One could also 
investigate the way in which old written notes and photographs from 
Heidi’s past are interpreted as signs of a changing climate and the death of 
Heidi’s sister in an accident when Heidi was a child. These notes and pho- 
tographs stand in stark contrast to the media products that Heidi and 
Kenneth produce together: they create pornographic films in order to 
make money for their CO,-neutral life. This peculiar media practice may 
be discussed while also participating in what Stacy Alaimo has discussed as 
independent, alternative and partly joyful activist outlets’ (porn- 
producers! ) way of engaging with the environmental debate (Alaimo 2016). 

After this brief intermedial analysis of the novel, it is now possible to 
discuss the novel in relation to the two questions: 


(a) How, from an intermedial point of view, do the media products 
represent scientific research related to global warming? 

(b) How, from an intermedial point of view, do the media products 
employ strategies that make hard science directly related to the 
everyday experience of non-experts in climate science? 


In response to (a), when it comes to representing scientific research in Den 
afskyelige, the aim of the novel is to represent and simplify the research at 
the same time to a level at which it can be understood by non-specialists. 

The main strategy of incorporating or representing scientific material in 
a work of fiction, a novel, is to embody it into the standard aspects of the 
narrative fiction. In this case, this takes place in at least two very typical 
ways. First, the entire storyworld of the imagined future world is informed 
by contemporary scientific theories and simulations concerning what the 
world will look like in a future affected by global warming. Second, many 
of the discussions of the protagonists relatively directly translate scientific 
material into actions, reflections, and direct conversations or discussions. 
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In this way, one of the typical medial aspects of scientific discourse—the 
attempt to produce a non-subjective discursive style that tries to express 
the non-subjective character of the scientific research—is directly opposed 
in this novel. Or, to formulate this differently, everything in a novel must 
be seen or described from the perspective of human beings, in this case 
Heidi, who is the narrator of the novel, and the characters of the fiction 
that she narrates. If the ideal of natural science is to produce knowledge 
that can be repeated by other researchers, who are usually anonymous, in 
controlled experiments or measurements, the golden rule of fiction is 
quite the opposite: to always embody everything that happens through 
the minds and bodies of narrators and/or characters; in fiction there is no 
attempt at objectivity in the scientific sense of the word. 

From this comes an answer to (b), my second research question: how 
do the media products employ strategies that make hard science directly 
related to the everyday experience of non-experts in climate science? What 
fiction does in its partly imagined, partly scientifically grounded story- 
worlds is to produce human figures who experience on their bodies and in 
their minds the more concrete but imagined conditions that are argued for 
by natural scientific articles. If the aim of natural scientific media products 
is to predict possible future worlds, in this case including the effects of 
global warming, the job of fictional storyworlds is to concretize and 
embody scientific research into very recognizable specific situations. In 
this way, the results produced in natural science becomes directly recog- 
nizable and understandable for readers who either do not wish to or are 
not able to read scientific articles. This means that the reader of Den afsky- 
elige is helped to experience, via the fictional storyworlds, what life might 
feel like in the future; Charlotte Weitze offers an interpretation of what she 
believes, according to research, to be future scenarios for a life in the 
Anthropocene. 


5.3.3 Comparison of Article and Novel 


Research Question 1 (Representation of Science) 

The comparison between the two texts follows the two research questions. 
The first research question has to do with the ways in which the two media 
products represent the scientific research results that in both cases consti- 
tute an important source behind the texts. 
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Here, the most important difference is probably that the CarbonBrief 
text directly paraphrases, refers to, and quotes science, whereas the novel 
has a much more indirect way of referring to the scientific research. In 
Weitze’s novel, science is constantly being put in the mouths of the pro- 
tagonists, in direct conversations and discussions and in more general ref- 
erences to ‘science says’ or ‘new research demonstrates’. To explain it in a 
different way, there is a basic narratological distinction between the texts: 
the CarbonBrief author functions as an explicit narrator who states the 
scientific facts, also by referring directly to quotes from scientists, but 
Weitze’s novel is organized in such a way that the scientific results and 
theories per definition are put in the mouths of persons inside the 
storyworld. 

Another clear difference is that the CarbonBrief article, starting with 
the discussion about actual weather phenomena, is particularly interested 
in making predictions about the future. Weitze’s novel, on the other hand, 
is set in such a future, but a fictive future; however, the people inside the 
storyworld do not, of course, experience the world as a future, but as their 
present living conditions and surroundings. 

This means, and this more or less sums up the points about direct/ 
indirect and future/present, that the popular science article, like science, 
aims at producing general notions about the ecological crisis and global 
warming. The novel, on the other hand, individualizes the scientific, gen- 
eralized notions ofthe future risks inherent in the Anthropocene condition. 

I have chosen two media products in two qualified media types that are 
both successful in many ways in what they are doing, at least if the criteria 
are their abilities to represent scientific results and theories in their respec- 
tive media forms. However, when comparing the communicative effi- 
ciency (regarding the article) or the aesthetic form and content (regarding 
the novel) criteria, it is clear that these could be more critically evaluated 
regarding their representation of science. The article could, for instance, 
be criticized for taking an unproblematized Eurocentric view on the 
Anthropocene future; this, presumably, has to do with the fact that British- 
based CarbonBrief, while having a good European and global outlook, 
nevertheless functions inside a distinct UK context, which explains the 
choice of first a photo from London and then the two European maps (as 
opposed to more global possibilities). Regarding a critical approach to the 
novel, it would be absurd to criticize the novel for only representing a 
limited, Eurocentric aspect of the global effects of the Anthropocene 
condition. 
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From a medium-specific perspective, a novel tends to be anchored in 
time and space almost by the logics of medium specificity: generally, the 
raison d’étre of fiction is to not speak in generalities but, on the contrary, 
to demonstrate how general social or scientific rules work on a human 
scale, that is, in the concrete lives of individuals or individuals as part of 
social groups. Therefore, if one has to criticize the novel, it should be on 
aesthetic grounds. A well-known aesthetic ideal is that when science (or 
philosophy or other external forms) ‘enters’ art it should do so in ways 
that mean that the science is introduced in the artistic universe in neces- 
sary or elegant or inconspicuous ways (for an example of this position as 
well as an historical discussion of the idea, see Gadamer 2013 [1960]). 
This is not always the case in Den afskyelige, where some conversations (or 
individual utterances in dialogues) sometimes read as almost direct quotes 
from newspapers or scientific journals without being properly incorpo- 
rated in the novelistic form as natural, spoken language uttered in a con- 
versation between two human beings. In these instances, the science is 
shoehorned into the fiction, so to speak: the author’s comprehensive 
research enters the novel as research and not as aesthetically motivated 
form and content. 


Research Question 2 (Scientific Results and Everyday Experience) 

It is of course impossible to completely keep apart the first research ques- 
tion (representation of science) from the second question, which has to do 
with how the two media products employ strategies that make science 
directly related to the everyday experience of non-experts in climate 
science. 

Communication, according to how it is normally defined, relates to the 
receivers of the communicative activity. Therefore, my admittedly slightly 
crude starting point was that one of the major targets, for both popular 
science communication and a mainstream novel focused on climate 
change, is transferring aspects of climate science results from hardcore sci- 
ence environments into new areas. Doing this also has to do with relating 
the scientific ideas to everyday experiences to a certain extent, and both 
the article and the novel clearly attempt to do this. 

Global warming, and most of the other effects of the critical ecological 
situation, are, as implied by their names, global in scope, and this is 
undoubtedly one of the reasons why relating these issues to specific, per- 
sonal experience is not always an easy task. Nevertheless, I have chosen 
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these two media products as examples of successful attempts to do so, and 
that is why my research does not ask if but how they do it. 

The article’s approach to this, as I have emphasized, is to base the 
global condition on a local phenomenon: the specific heatwaves described 
in the text and represented by photography are used as signs of a global 
warming tendency. So, the heatwaves, which many people in Europe have 
relatively fresh memories of, were not only problematic for one person 
(the person in the photo) but also for many others who can easily identify 
with him, and supposedly for the entire European population who are, as 
shown via the maps of Europe, experiencing this warming. The man in the 
photograph and the maps are characterized by anonymizing—the idea is 
that anybody will be able to identify with the man and the nations, and the 
goal of the article is to make the general point, most of all on a cognitive 
level; for all we know, the man might be an actor and the maps are 
extremely general. 

Again, the novel’s strategy is distinctly different: it offers ways of very 
directly understanding what life might be like in a warmer world, not in 
general terms, but by way of identifying fully with specific persons with 
names, a fictional background, bodies, families, and jobs. Presumably, the 
intention of the author is that such a concretized, specific description is a 
powerful way to make readers understand how it feels (in both the psycho- 
logical and the sensorial sense of the word) to live in such an Anthropocene 
future. Needless to say, the pattern from the first research question returns 
here. It is a question of the generality of science versus the specificity of 
arts and literature. 


5.4 CONCLUDING REMARKS 


My impetus for writing this article is far from original: it stems from the 
well-known observation that the natural sciences produce knowledge and 
data which are absolutely crucial for the world to know about and act 
upon—but that the transfer of scientific knowledge to the general public 
is neither unproblematic nor simple. From this basic fact followed an iden- 
tification of a gap in the broad field of environmental humanities concern- 
ing this question, and also in the social scientific fields investigating, for 
instance, risk, health, and science communication. As far as I have been 
able to establish, these fields have not developed the abstract theoretical 
ideas or the specific methods to deal with the plethora of different specific 
media products or the more generic qualified media types that deal with 
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these questions. I have not come across a scientific approach which is able 
to effectively handle the problems involved in describing, analysing, inter- 
preting, and finally comparing radically different media products from 
completely different media backgrounds, even if they are representing 
clearly comparable scientific content. 

Having first identified this problem, I then tried to suggest a theory and 
method called intermedial ecocriticism which, as the name suggests, com- 
bines two different research fields and traditions, namely intermedial stud- 
ies and ecocriticism. Because it is part of environmental humanities, one of 
the main strengths of ecocriticsm is its deep understanding of the histori- 
cal aspects and the critical level of the ecological crisis. As a research field, 
intermedial studies, on the other hand, has a thorough understanding of 
the problems and complexity of representation in general, and, in particu- 
lar, intermedial studies has developed tools to analyse comparable content 
and forms in different media. 

I tried to sketch out briefly some of the most important aspects of the 
two scientific fields before I then ventured to propose an analytical method 
that borrows, so to speak, the focus on ecological issues from ecocriticism 
and the focus on representational issues across media from intermedial 
studies. I suggested analysing and comparing ecomedia products in two 
consecutive analytical steps, which led to a comparison of, in this case, an 
online popular scientific article and a novel. I tried to conduct the com- 
parison by establishing two specific questions, one regarding how the 
media products represent scientific research regarding climate change, and 
the other regarding how the ecomedia products relate the scientific results 
to the everyday non-specialist experience. 

When doing comparative analysis across media, it is tempting to end up 
on an evaluative or even competitive note: which media product is *best? 
at doing x or y? Given the fact that what I refer to as ‘weak’ or ‘temporary’ 
medium specificity does exist, such questions are not uninteresting, and it 
is easy to see that some media are, in general terms, more suited to repre- 
sent certain things than other media. It would, for instance, be rather dif- 
ficult (but not totally impossible) for the otherwise highly developed and 
utterly sophisticated medium of symphonic classical orchestral music to 
express, clearly and unambiguously, the three major changes made in the 
state budget of the Swedish state from 2018 to 2019—whereas that would 
be relatively easy to do in a short, written journal article in a daily 
newspaper. 
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In the case study examples that I had selected for my investigation, I 
was not really able to identify such obvious differences in the ‘success’ of 
representing the two issues I had focused on. That might be because I had 
chosen two media products that were in fact relatively similar (they are 
both examples of narratively ordered material in which the predominant 
basic modality is written language), but that was not really what I had 
expected to accomplish with my analytical method. Instead, what the 
method was able to detect were some relative but not very dramatic advan- 
tages accomplished by, respectively, the science journalist Daisy Dunne 
and the fiction writer Charlotte Weitze; these advantages were to a large 
degree dictated by the ‘weak’ medium-specific affordances of the qualified 
media they worked in. 

It became clear that both Dunne and Weitze had obviously worked 
intensively to incorporate natural scientific research into their texts. The 
main difference was the way in which the article tries to refer directly and 
unambiguously to ongoing scientific research and sources, whereas the 
novel needs to transfer the scientific research from direct references into 
either (a) a general construction of a future, fictional world dominated by 
global warming or (b) speech in the mouths of the protagonists when they 
discuss how to live in such a world. From a superficial point of view, the 
popular scientific article perhaps could be thought to be much better 
suited to doing this, but I have tried to demonstrate that Weitze has man- 
aged to include quite a lot of scientific material in her novel (as was clear 
from her mentioning numerous scientific expert sources at the end, too). 

When it comes to relating the science to everyday experience (my sec- 
ond question), the novel, following conventional wisdom, probably 
‘ought’ to do best at this. By working with fictional persons in recogniz- 
able but fictional surroundings, it should be easy for a novelist to make 
science matter to non-scientists. Weitze did not fail at this task, but even 
the journalist Dunne shrewdly used the medium-specific available means 
of the online popular science article to construct a small narrative consist- 
ing of a man in the not-forgotten local heatwave of 2018 in London that 
circled outwards to a European climate condition that was easy for many 
readers to identify with. Banalizing my subject a little bit, I conclude that 
it was a draw between Weitze’s somewhat surreal cli-fi novel and Dunne’s 
narratively ordered popular science article. 

However, even if this competition ends in a draw, I do hope that per- 
haps the idea of an intermedial ecocriticism might be a winning concept. I 
maintain that we need tools to analyse and discuss the scientific research 
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regarding our epoch’s most alarming topics, and even if we in the humani- 
ties cannot conduct the natural scientific research ourselves, we can cer- 
tainly contribute to the thinking about the ecological crisis in crucial 
ways—in particular by investigating all the different media products, in all 
the different media types, that do the difficult work of transporting scien- 
tific material across the media borders to where most people learn what 
the sciences are doing. 


NOTES 


l. For a recent literature review of some of these fields, see Chadwick (2017). 

2. Many individual media types outside literature have been under ecocritical 
(in the broad sense of the word) scrutiny, for instance, film studies (Kaplan 
2015), visual art (Brenthel 2016; Demos 2017), music (Allen and Dawe 
2016; Hart 2018), and design (Sherin 2013)—to mention just some of the 
more recent publications in a rapidly expanding field. 

3. In earlier publications, I have used a general distinction between media 
combination and media transformation, but here I follow Ellestróm's new 
terminology (2020). 

4. I have critically discussed Morton's notion of hyperobjects in Bruhn (2018). 
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6.l INTRODUCTION 


In this chapter, I attempt to test the applicability of Lars Ellestróm's model 
of intermedial relations (2020) to analyses of the device of narrative 
metalepsis in a number of different media products stemming from two 
different media types: literature and film. At the same time, the chapter 
has the more ambitious aim to illustrate certain aspects of the model itself 
using the concept of metalepsis. The overarching aim is to highlight the 
transmediality of the device of metalepsis, that is, its potential to “be 
employed in various media" (Rajewsky 2002: 13). As such, the chapter is 
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an attempt to break with what Lars Elleström sees as one problematic 
tendency in today's intermedial studies, that is, that “/m Jedia in general 
are studied through concepts developed for language analysis” (Ellestróm 
2020: 6). Metalepsis is a good example in such an endeavour, since its 
definition was coined within the frame of a linguistic medium, namely lit- 
erature. This is, I argue, no reason to persist in studying metalepsis from a 
language-based perspective, which is the tendency even in the numerous 
cases of approaching metalepsis in other media types than linguistic ones. 
It is, I argue, not sufficient to acknowledge the transmedial potential of 
metalepsis. The device should also be seen completely beyond its initial 
connection to the language-based context, which is best done by using 
Ellestróm's model of intermedial relations. 

The initial definition of metalepsis is, as mentioned, clearly linked to 
language, especially written language, in the form of the qualified medium 
of literature. Indeed, French narratologist Gérard Genette gave the first 
definition of metalepsis as “any intrusion by the extradiegetic narrator or 
narratee into the diegetic universe (or by the diegetic characters into a 
metadiegetic universe, etc.), or the inverse” (Genette 1980 [1972]: 
234-235). The all-importance of the narrator is something that all subse- 
quent studies have had a hard time avoiding, even in the case of media 
types where the existence ofa narrator is not necessary, or even impossible. 
Indeed, theorists explain metalepses in narratorless media, such as painting 
or music by analogies to literature, finding that certain devices are ways to 
imitate a narrative voice where such a thing cannot exist. Even if this is 
true in many individual cases, it is my contention that it should not be 
transformed into a rule. Indeed, narrative levels can be both constructed 
and transgressed without the use of a narrative voice, as I will try to show 
in the chapter. 

Interestingly enough, the transmediality of the concept of metalepsis 
was alluded to already in Genette's first theorizations in 1972. First, he 
explicitly illustrated metalepsis with examples from other media types than 
literature—such as theatre, paintings (Genette 1980 [1972]: 235) and 
cinema (1980 [1972]: 237). Then, implicitly, when he used the term 
‘world’ in his other definition of the concept as a transgression of “the 
sacred frontier between two worlds, the world in which one tells, the 
world of which one tells” (1980 [1972]: 236), he suggested the extension 
of the concept of narration beyond its linguistic frame and its inclusion 
into the broader spectrum of representation. A decade after the first defi- 
nition, Genette emphasized the importance of representation for metalep- 
sis even more strongly, avoiding the reference to a narrator, and defining 
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it as the “deliberate transgression of the threshold of embedding” (Genette 
1988 [1983]: 88). As Jean-Marie Schaeffer pointed out, this redefinition 
permitted the transfer of the concept to the field of representation in gen- 
eral (2005: 325, 327). 

This expansion of metalepsis beyond only language-based media to the 
field of representation in general should have led to a liberation of the 
concept of its linguistic straitjacket. Unfortunately, this has not really hap- 
pened. As in Genette's early definition, where the act of telling is still 
central even when a certain move towards representation is made, the 
linguistic basis persisted both in definitions and in analyses. This is, for 
example, very clearly the case for Sabine Lang's detailed typology of the 
so-called paradoxical narrative devices, which includes metalepsis as the 
most conspicuous case, where the main criterion is the difference between 
story and discourse (discurso and historia in the Spanish original, Lang 
2006: 32). This is, to an even higher degree, the case for Sonja Klimek's 
approach, when she claims that narrative levels only concern *written and 
not visual fiction? (Klimek and Kukkonen 2011: 25). This is also the case, 
admittedly to a lesser degree, even in a number of studies, from Brian 
McHale’s Postmodernist Fiction (1987) to more recent ones such as 
Werner Wolf’s *Metalepsis as a transgeneric and transmedial phenome- 
non" (2005), the anthology Metalepsis in Popular Culture edited by Sonja 
Klimek and Karin Kukkonen (2011), Jan Alber's and Alice Bell’s 
“Ontological metalepsis and unnatural narratology" (2012) or Jeff Thoss's 
When Storyworlds Collide: Metalepsis in Popular Fiction, Film and Comics 
(2015). These studies all have in common the aim to expand the applica- 
tion field for metalepsis to representational media in general. However, 
despite the opening to other media, the studies are still dependent on the 
concept’s initial connection to language, and have the tendency, criticized 
by Ellestróm, to compare only two media types, out of which literature is 
always the basis (2020: 5-6). 

It might seem that even the current study does the same thing. 
However, even if only two media types are studied, the point is to consider 
the media beyond their essence, and concentrate on their modalities. This 
is, I argue, a more productive way of approaching transmediality than 
studying many different media, because no matter how many media one 
might study, there would always remain cases that could contradict the 
results. Looking at a medium's basic ingredients, which is the aim of 
Ellestróm's model of intermedial relations, is a way out from this dilemma. 
It also helps in expanding metalepsis beyond language to representation in 
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general, since representation is the core of what Ellestróm calls the semi- 
otic modality. 

According to Ellestróm's model, each medium consists of four elemen- 
tary modalities: the material, the sensorial, the spatiotemporal and the 
semiotic. These modalities form a transmedial basis on which more com- 
plex features can be constructed. These features, that Ellestróm first called 
“compound media characteristics” (2014) and now changed to “repre- 
sented media characteristics" or simply *media characteristics" (2020: 
80), cannot be transmedial in the same way as the elementary modalities, 
since indeed they are more specific, and also more or less closely linked to 
a specific medium. This can be explained in a simple way by the device of 
the subjective camera, which is a media characteristic linked to a medium 
where a camera is necessary. Still, at a level below, the level of elementary 
modality, a subjective camera has to do with the sensorial modality, more 
exactly with the visual mode, which belongs to the transmedial basis. It is 
exactly by acknowledging such properties of lower complexity that one 
can discover certain similarities between devices from different media 
which seem completely incompatible at a first look. 

Paradoxical narrative devices such as metalepsis are examples of repre- 
sented or compound media characteristics. As such, they could hardly be 
considered as fully transmedial. Not even narration in general can be con- 
sidered to be a trait that can be found in all media, despite the so called 
narrative turn, according to which everything is narrative (see, for instance, 
Richardson 2000 and Hyvarinen 2006). But this is not a good reason to 
reduce the applicability of the paradoxical narrative devices to only one 
qualified medium: literature. The main reason is the fact that narration can 
hardly be seen as only a literary trait. The other reason is, as I already men- 
tioned, that metalepsis is linked to representation in general rather than to 
narration through language only. 

Still, as already mentioned, there is a tendency even today to concen- 
trate on two media-specific aspects of metalepsis, which both have reduc- 
tive effects. First of all, there is the language-based aspect, especially 
highlighted in the insistence on the need of a narratorial voice for the 
metalepsis to occur. Secondly, there is the fictional aspect, since several 
theorists find that metalepsis can only occur in fiction. Some even find 
metalepsis to be *a popular candidate for a fiction specific category" 
(Bareis 2008: 164). 

I argue that both these aspects can be approached more productively by 
using Ellestróm's model of intermedial relations (2020). Indeed, both 
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narratorial voice and fictionality are media characteristics that can be stud- 
ied in relation to the basic modalities of the media types in which metalep- 
sis occurs, rather than being considered as necessary ingredients for the 
concept of metalepsis. I will approach both these aspects below, on the 
basis of a number of examples from two media types. 


6.2 . METALEPSIS IN LITERATURE 


I start with an example of metalepsis in the qualified medium where its 
existence is undisputed: literature. The example is, moreover, one of the 
best known cases of metalepsis, and has been often analysed as such by 
theorists as Dorrit Cohn (2005: 123), Sabine Schlickers (2005: 161), 
Sabine Lang (2006) and Bénédicte Vauthier (1999). It comes from 
Spanish author Miguel de Unamuno's novel Mist ( Niebla in original), 
written in 1907 but published only in 1914 (2013 [1914]. 

At first view, the novel seems to be realistic, even though there is a cer- 
tain play with the narrative instance at the beginning. Indeed, the first 
pages are narrated by a certain Victor Goti, in the first person. However, 
as it appears after these pages, this Victor Goti is not the main narrator. 
Goti only supposedly writes a prologue to the novel we are reading, and 
contradicts some of the facts therein, pretending to know the characters 
better than Unamuno. Unamuno himself responds to this prologue, in a 
post-prologue, defending himself against Victor Goti's accusations, before 
letting a heterodiegetic narrator (who will later prove to be himself) take 
over the narration. This narrator tells the story of a certain Augusto Pérez, 
a rich young man without a clear goal in life. After his mother's death, 
Augusto falls in love, or rather thinks that he falls in love, with Eugenia, a 
young pianist of lower social status. Eugenia, however, does not love 
Augusto, but runs away from him with a young man of even lower social 
status. Having even lost this last goal in life, Augusto decides to commit 
suicide, but not before going to Salamanca to discuss the matter with a 
certain Miguel de Unamuno, who had written an essay on the question of 
suicide from a philosophical point of view. 

It is this dialogue with Unamuno that constitutes the famous metalep- 
sis in this novel. Indeed, during their talk, which takes place in chapter 31 
out of 33, Augusto finds out that he is only a character of fiction created 
by his interlocutor. Unamuno reveals to Augusto that he is destined to die, 
not to commit suicide, as he was planning. Augusto is shocked at first, but 
then starts to contradict Unamuno, questioning whether he too exists in 
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the real world. Something that we, as readers, could ponder upon too, 
since the Unamuno of the story is hardly the same Unamuno as the one in 
the real world. Augusto leaves Salamanca after the dialogue in a very hasty 
manner, arrives at home and discusses his fictional status with his servants, 
who do not understand the matter. Eventually, Augusto dies, but it is not 
stated clearly if it is through suicide by eating compulsively or if it is by 
Unamuno's narratorial decision. 

In chapter 31, the narrator abandons his heterodiegetic status and 
becomes homodiegetic, meaning that he narrates at the first person, as a 
character in the story. This is how this transition is done, at the beginning 
of the chapter: *The storm in Augusto's soul ended in a terrible calm. He 
had resolved to kill himself, to put an end to that self which had been the 
cause of all his misery. But before carrying out his plan it occurred to him, 
like a drowning sailor who grasps at a near plank, to come and talk it over 
with me, the author of this whole story" (Unamuno 2013 [1914]: 147, my 
emphasis). Or rather, I would argue, the narrator of the story, since the 
character of Unamuno is fictional too, as I already suggested. 

Admittedly, the metalepsis here respects Genette’s definition, since the 
frontier between the world in which one tells and the world of which one 
tells is clearly broken. However, it is interesting to notice that it is not until 
this chapter that the narrator appears as a character in the storyworld. 
Indeed, not only does he use the first person when speaking, but he is 
represented as a character, who physically meets Augusto. It is what theo- 
reticians, such as Marie-Laure Ryan (2006: 207) and Monika Fludernik 
(2005: 79), classify as an ontological metalepsis, the most conspicuous 
and shocking example of the different classes of metalepsis. What I con- 
sider noteworthy, however, is that it seems that even in the case of such a 
typically literary device, there is a need for a kind of representation that 
goes beyond the strictly literary context. What I mean is that the narrative 
levels are less conspicuously transgressed through the literary device of the 
voice of the narrator. They are more clearly transgressed by a represented 
physical meeting between two characters that are not supposed to exist in 
the same world. It is true that the meeting becomes all the more shocking 
thanks to the narrator’s using the first person, and referring to the story as 
his own creation, but such a meeting could be described without the use 
of a first person narrator, just by letting the different worlds collide. The 
only requirement is that the worlds should be at different, incompatible 
narrative levels. 
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It is probably the requirement of ‘narrative levels’ that links metalepsis 
so closely to language. But, as I already mentioned, narration can be real- 
ized through other means than just language. And indeed, Genette’s defi- 
nition of the two worlds on the basis of ‘telling’ seems to be an unfortunate 
slip of the tongue. What should be at stake are the levels of representation, 
and metalepsis should be the violation of the frontier between different 
levels of representation. Neither the definition based on the notion of a 
narrator (I recall: “intrusion by the extradiegetic narrator or narratee into 
the diegetic universe”) is particularly relevant, since the violation can be 
achieved by other means than the narrator. Just to mention one example, 
an object that moves from one narrative level to another narrative level 
would be an undoubted case of metalepsis too. 

What is particularly interesting is that the language-based aspect of 
metalepsis seems to be hard to get rid of even in other media. One exam- 
ple is the cinematic adaptation of Unamuno’s novel, made by Spanish 
director Fernando Méndez-Leite in 1975. The meeting between Unamuno 
and Augusto is represented in the film too, but Unamuno is heard as a 
narrator in voice-over, addressing the viewers and explaining that the 
meeting is between him, as author, and a character created by him. The 
words are very close to the text of the novel. However, two additional 
devices typical of the cinematographic medium are used. The first one is 
the editing technique, used here in such a way as to give the viewer the 
impression that Augusto appears and then disappears from Unamuno’s 
room out of nowhere, almost in a supernatural way. It is as if Unamuno 
created him right there and then in front of us, the viewers. This is a very 
good example of how other means than strictly linguistic, in this case 
visual, can be used in order to create a metaleptical effect. The question is, 
however, whether the effect would have been metaleptical if the viewer 
had not been aware of the different narrative levels established previously, 
in a language-based way, by the voice-over. I argue that even if it is not 
done in the film, such an establishment could have been done by other 
means, and choosing to repeat the narrator’s words from the novel is 
probably the director’s choice to show his debt to the original media prod- 
uct, by incorporating the part that can be incorporated in the new medium. 
Admittedly, to use Ellestróm's words, the text has been transmediated 
(2014), from written to oral, but it is still a reference to the original work. 
The second cinematographic device is the use of sound effects. There is a 
certain change in the quality of the sound directly before Augusto's 
appearance and until his disappearance, clearly perceptible for the viewer, 
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who gets the impression that the narrative voice changes its status from 
being intradiegetic to being extradiegetic—that is, addressing the viewer 
directly, as a voice-over. It is still the same voice, but it is as if it came from 
another place and sounding as if reading out loud from a novel. Here, I 
argue, the change is solely done by devices belonging to the mode of 
sound. A narrator’s voice is imitated in its aural characteristic, not claimed 
by words. It is something that cannot be done in a novel, where the sound 
as mode is simply absent. 

This whole discussion around the ways the metaleptical encounter 
between the narrator and the characters created by him is represented can 
seem rather academic. As a matter of fact, even if we can conclude that 
there is only one narrative level in the scene, since the narrator is actually 
just a represented character, this does not mean that there is not a meta- 
lepsis there. Actually, the metalepsis had already occurred before the 
encounter, when the author fictionalized himself, moving from the 
extradiegetic level to the intradiegetic level. This is shown more clearly by 
the cinematographic media product, where it is much more important to 
expose visually the moment of this diegetic change so that everything 
becomes more comprehensible. 

Beyond this rather academic analysis, the question is why metalepsis is 
used. The question is relevant at a general level, since metalepsis is a device 
which points to the artificiality of a media product, so it should rather be 
avoided in films aiming at realism. In the specific case of Niebla, one 
answer is suggested in the novel itself: it could be that becoming a charac- 
ter in a literary work is a way to immortalize oneself. And indeed, it is a 
strategy that seems to work. A little more than a century after its publica- 
tion, Unamuno is, in a certain way, paradoxically present in the midst of 
his readers. But is that a result of his transformation into a character in his 
own book? Is it not rather the result of the success of the books? Beyond 
these questions, there is something particularly interesting in the way in 
which reality and fiction meet in such a paradoxical way. What happens is 
that a confusion is created at different levels. For instance, is it the author 
who moves towards the inferior narrative level—metalepsis—or is it the 
character, Augusto Pérez, who moves towards the superior level—anti- 
metalepsis according to Genette's definition? Indeed, Augusto is the one 
who decides to visit Unamuno. It is true that this decision is ultimately 
orchestrated by Unamuno, as it is underlined by his lack of surprise and 
enigmatic smile when Augusto appears in his office (2013 [1914]: 147). 
However, the development of the situation eventually surprises Unamuno, 
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first when Augusto tells him that *you are nothing more than a pretext for 
bringing my history into the world” (2013 [1914]: 149), then when he 
acknowledges his reaction: *These sallies of Augusto were beginning to 
make me uneasy? (2013 [1914]: 151). The gradual loss of authority over 
his own creation complicates things, putting into question whether the 
details are planned by the author or not. 

Before this dialogue, the narrator's authority had not been put into 
question at all, with the exception of the already mentioned prologue. 
Even then, Victor's words had not really questioned Unamuno's authority 
in general; they only questioned his credibility concerning the version of 
the facts. On the contrary, the prologue establishes the extradiegetic status 
of the narrator, Unamuno, which is reinforced in chapter 31, when the 
narrator refers to himselfas *me, the author ofthis whole story? (Unamuno 
2013 [1914]: 147). Thanks to this clear distinction between the different 
narrative levels, the metalepsis is very effective in Niebla. The fusion of the 
extradiegetic and the intradiegetic levels in chapter 31 has indeed a strong 
effect on a reader who until then was certain of being able to keep the 
levels apart. This gives also the opportunity to double entendres in the 
novel, such as the one when Augusto tells his servant: “I come from 
another world, Liduvina, and I am going to another world? (2013 [1914]: 
155). This other world could, indeed, be the world of the dead, but also 
the world of the author, the extradiegetic, real world. This double enten- 
dre, besides being a game, can be a way to underline the difficulty to dis- 
tinguish between fiction and life, between dream and reality. 

In conclusion, one of the main elements of Niebla is the fusion between 
different narrative levels, especially between the extradiegetic level of the 
author and the diegetic level of the characters. Moreover, Niebla antici- 
pates what Genette would define as one of the possible effects of metalep- 
sis: the existential doubt. Indeed, Augusto comments upon the meeting 
with his creator by making an analogy between the act of writing and 
God's act of creation. The result could be that the real Unamuno could in 
his turn be a fictional creature, the product of God's dreams. This is actu- 
ally suggested by Augusto: *Very well, then, my lord creator Don Miguel", 
he says, using an ambiguous “lord creator" that could refer to God, “you 
too are to die, you too! And you will return to that nothing of which you 
came! God will stop dreaming you!” (2013 [1914]: 154). Genette's 
words about the existential doubt that can be created by metalepsis are 
close to Augusto’s: “The most troubling thing about metalepsis indeed 
lies in this unacceptable and insistent hypothesis that the extradiegetic is 
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perhaps always diegetic and that the narrator and his narratees—you and 
I—perhaps belong to some narrative” (Genette 1980 [1972]: 236). In the 
following parts, I will investigate whether metalepsis can be achieved in 
similar ways in a media type where language is less central, that is, film, to 
see whether the same effects can be created using other media 
characteristics. 


6.3 METALEPSIS IN FILM 


One of the reasons why I choose to analyse metalepsis in film is that film 
consists of complex modes which can be combined in creative ways. The 
material modality of film is, for instance, flexible enough, as we shall see, 
to incorporate other media types, or to give the impression of incorporat- 
ing them. For instance, since the basic medium of film consists of moving 
images, it can easily incorporate still images. And since film has a sound 
track, it can easily incorporate any media type that is based on sound, such 
as music, radio, and so forth. Thus, I can analyse the effects that other 
modes, which are less important in literature, can have on metalepsis. 

Another important reason why film is a suitable study object in this 
context is that it has a very special relationship to literature. This is not 
only because film, as a qualified medium, has a strong tradition of adapting 
literary works (which is a form of transmediation, to use Ellestróm's words 
[2014 ]), but also because film, as I just mentioned, can incorporate writ- 
ten text. For instance, since the visual mode is of great importance in film, 
a written text can be shown on the screen, (which is a form of media rep- 
resentation, according to Ellestróm's model [2014]). Moreover, film can 
imitate more complex literary media characteristics in very interesting 
ways. 'The use of a narrator in voice-over, as we saw in the adaptation of 
Unamuno's novel Niebla, is just one such example, even if not the most 
interesting one, since it is quite obvious. In intermedial theory, this is what 
Werner Wolf would call *intermedial imitation"? and Irina Rajewsky *inter- 
medial reference”. It is, as Wolf puts it, when “the signifiers of the work 
and/or its structure are affected by the non-dominant medium, since they 
appear to imitate its quality or structure? (2002: 25), or, as Rajewsky puts 
it, when *the given media-product thematizes, evokes, or imitates ele- 
ments or structures of another, conventionally distinct medium” 
(2005: 53). 

The adaptation of Niebla is illustrative as an example of the difficulty of 
completely breaking with the literary model, in the case of metalepsis. As 
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I already mentioned, I argue that a narrator’s voice-over was not actually 
needed in order to prepare the field for the metalepsis to occur and to be 
understood as such by the public. Meanwhile, it is understandable that the 
director imitates certain details from a specific source media product (the 
specific novel by Miguel de Unamuno) when adapting it to another media 
type. It is also understandable that metalepsis, being a very complex media 
characteristic, cannot be imitated without imitating its smaller units, such 
as the narrator’s voice. What is less understandable meanwhile is that many 
films imitate a narrator’s voice as a general feature (that is not from a spe- 
cific novel) in order to achieve metalepsis. I will present here two such 
cases, Stranger than Fiction (2006) and Le tableau ( The Painting) (2011), 
but I will also go beyond the narrator’s voice and analyse whether other 
filmic media characteristics are used when staging the metalepses. By 
doing so, I aim at showing that metalepsis is not necessarily dependent of 
a narrator’s voice. The third case is an example of a film, Woman at War 
(2018), where the narrator’s voice is not used at all in the creation of 
metalepsis. 


6.3.1 | Metalepsis Through Narrative Voice in Film 


The first film I choose to analyse is Marc Forster's Stranger than Fiction 
(2006). As the title suggested, the play with the relationship between fic- 
tion and reality is the main theme of this film, and is achieved through a 
metalepsis which actually reminds of the one in Niebla. Indeed, the main 
character, Harold Crick, played by Will Ferrell, hears the voice of a narra- 
tor, played by Emma Thompson, who seems to create the events he is 
experiencing, and that the viewers are viewing on the screen. The viewers 
hear the narrator's voice-over as well, and have no reason to consider this 
strange at all. Indeed, this is a case of an intermedial imitation of a narra- 
tor's voice, common for many films. The first time the viewer reacts is 
when the character himself reacts, very early in the film, when starting to 
hear the voice, which describes him and his ongoing actions. 
Interestingly enough, what the metalepsis actually does is to draw the 
attention to the artificiality of the device of a narratorial voice-over. What 
in literature is unavoidable, that is, the use of a narrator narrating the 
events, is unnatural in film, and has become an accepted device only 
because of its frequent use. The existence of an extradiegetic voice at the 
same level as the storyworld is actually a logical impossibility, since the two 
worlds should not be able to coexist. The ability to keep the worlds apart, 
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even if they are represented as coexisting, is only achieved through train- 
ing, and a viewer who experiences such a case for the first time would 
probably react with the same surprise as does Harold. 

The similarities with Nzebla do not finish at this point. Indeed, not only 
does the character have a physical contact with his creator, but he also 
hears that the creator plans his death, as Unamuno did for Augusto. 
Harold reacts at first as does Augusto, trying to change the course of the 
events. For instance, he spends all his time at home trying to control his 
destiny by doing nothing and thus not taking any risks, but even so, he is 
almost killed by a wrecking crew which demolishes his apartment, because 
of having mistaken the building for an abandoned one. Having thus real- 
ized that he cannot control the plot, Harold accepts his fate and tries to 
enjoy the time he has left. One day, by coincidence, Harold hears the voice 
he had in his head when watching a television interview with author Karen 
Eiffel. Indeed, the voice is the same as the narrator’s, but as a character. 
When confronted by Harold, who did his best to find her, Karen Eiffel is 
not aware of what she had caused. She is just a character in the story, but 
in a strange loop, it is the story she writes. Therefore, she is shocked too. 

It is certainly not by chance that a British actor has been chosen for the 
role of Karen Eiffel. First of all, the British accent of the narrator’s voice 
contrasts in an obvious way with the American accents of the other char- 
acters, who are supposed to live in Chicago. This contrast contributes to 
creating the impression that the voice is heard at another narrative level, 
the extradiegetic one, that is, the level that should not be heard by the 
characters from the diegesis. The fact that the voice had been disembodied 
until the author first appeared on the TV set also contributed to this 
impression, as did the sound effects. Besides, also by effect of contrast, 
British accent can give an elevated impression, which can be considered as 
a reference to literature. All these devices are filmic media characteristics, 
relying on the mode of sound, and could thus not be used in written lit- 
erature, where the mode of sound is absent. It is therefore an interesting 
example of how intermedial imitation can be used in innovative ways, thus 
casting new light on concepts that previously were considered to belong 
to literature. 

When Karen Eiffel, the author, appears as a represented character in the 
film, her accent is still British, and her voice is the same, but it loses its 
distinctiveness. This is, I argue, the result of the viewers being used to the 
filmic conventions. A voice belonging to a character who is visually repre- 
sented is less distinctive than a disembodied voice-over. Even if the 
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voice-over is known by the viewers to belong to one character, not show- 
ing the character and using certain sound effects can easily give the impres- 
sion that the utterance comes from another level at the moment it is made. 
That is why the first sign of metalepsis, when Harold actually hears this 
strange voice, is so shocking for both Harold and the viewers. 

The second example of metalepsis in film that I choose to analyse is 
Jean-François Laguionie’s Le tableau (The Painting) from 2011. It is an 
animated film which can be considered as a metalepsis from the start until 
the end, since its first scene is a metaleptic entrance into the world of film, 
where the viewer stays until the last scene. Besides, the film is a very good 
example of different types of metalepses, according to classifications estab- 
lished by theoreticians in the field, showing how film as basic medium can 
contribute to a better understanding of these classes and of metalepsis in 
general. 

The most important distinction the film addresses is the one between 
vertical metalepsis and horizontal metalepsis. This is a distinction first 
made by Klaus Meyer-Minnemann and Sabine Schlickers (2005) and by 
Sabine Lang (2006). Their definition of vertical metalepsis as a transgres- 
sion of the border between worlds situated at different narrative levels (the 
world of the narrator and the world of the narrated events) is close to 
Genette’s initial definition, that is, “any intrusion by the extradiegetic nar- 
rator or narratee into the diegetic universe” (Genette 1988 [1983]: 
234-235). Horizontal metalepsis, defined as the transgression of the bor- 
ders between worlds situated at the same narrative level (Meyer- 
Minnemann and Schlickers 2010: 140), is an expansion, rather than a 
contradiction, of Genette's definition. Still, the concept of the horizontal 
metalepsis has raised certain controversy among theoreticians. Sonja 
Klimek considers, for instance, that we should “respect Genette's initial 
definition? and restrict the use of metalepsis only to cases where there is a 
vertical infringement, that is, between “the world of the creator” and “the 
world that is created within the artefact” (Klimek 2011: 26). My analysis 
of the metalepses in Le tableau will try to show how cinematic medium 
can help to reconcile these views. 

When I claimed above that the whole film is a metalepsis, it was because 
it is framed by two vertical metalepses, which form the entrance into and 
the exit from the world of the film. The whole story thus becomes the 
result of the initial metalepsis, and its fictional status is regularly reminded 
by metaleptical comments or by metaleptical scenes. The first scene is not 
only worth studying as an interesting case of a vertical metalepsis, but also 
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because it occurs both visually and verbally. Indeed, while the camera is 
zooming in, giving the viewer the impression that the frame is left behind 
and that the limit established by the canvas is transgressed, a character in 
the painting, a girl called Lola, comes to life and addresses the viewers with 
the words: “Voilà ! Vous venez de pénétrer dans le tableau” (“Here you 
are! You just entered the painting”). Such a comment could be interpreted 
as a metalepsis of the discursive kind according to Dorrit Cohn (“méta- 
lepse discursive” in the French original), who found it more “harmless” 
(“inoffensive” in the French original) than the “diegetic metalepsis” 
(“métalepse de Phistoire” in the French original) (Cohn 2005: 122). John 
Pier would consider this example a “minimal metalepsis” (“métalepse 
minimale” in the French original) (Pier 2005: 249), considering that the 
transgression of the frontiers is only suggested in such a case, not real. 
Monika Fludernik would classify it as “rhetorical metalepsis” (2005: 
79-81) as opposed to the “ontological metalepsis”, according to a classi- 
fication made previously by Marie-Laure Ryan. According to Ryan, onto- 
logical metalepsis “opens a passage between levels that result in their 
interpenetration, or mutual contamination,” while rhetorical metalepsis 
only “opens a small window that allows a quick glance across levels, but 
the window closes after a few sentences, and the operation ends up reas- 
serting the existence of the boundaries” (Ryan 2006: 207). Indeed, the 
comment is quite close to the example these theorists use to illustrate 
these kinds of harmless metalepses, namely an extract from Honoré de 
Balzac’s Les illusions perdues where the narrator addresses the narratee 
with the words: “While the venerable churchman climbs the ramps of the 
Angouleme, it is not useless to explain the network of interests into which 
he was going to set foot” (quoted, for instance, by Fludernik [2005 ]). In 
the French original: “Pendant que le vénérable ecclésiastique monte les 
rampes d'Angouléme, il n'est pas inutile d'expliquer le lacis d'intéréts dans 
lequel il allait mettre le pied.”). An even better example, also mentioned 
by theorists, comes from another well-known French nineteenth-century 
novel, Alexandre Dumas's The Count of Monte Cristo: 


We will leave Villefort on the road to Paris, travelling—thanks to trebled 
fees—with all speed, and passing through two or three apartments, enter at 
the Tuileries the little room with the arched window, so well known as hav- 
ing been the favorite closet of Napoleon and Louis XVIII., and now of 
Louis Philippe. (Dumas 2019 [1844]: chapter 10) 
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What complicates things in the film, however, is the visually represented 
‘penetration’ of the painting. It could be argued that this is only a meta- 
phor of the act of immersion into the painting, since we as readers or view- 
ers penetrate in some way the world represented by the media product. 
But this interpretation is a part of the semiotic modality. At a lower level, 
the spatiotemporal modality, the fact remains that the zooming in of the 
camera gives the viewer the impression of entering a space, which is only a 
virtual space, represented on the screen. Indeed, neither the painting nor 
the space represented by the painting exists in reality. Still, by using visual 
devices offered by the cinematic medium, the viewer gets the impression 
of transgressing the two borders: first the border between reality and the 
world of Lola as narrator, and then the border between that world and the 
world of the painting. This second border is even more clearly marked by 
the frame of the painting. 

My point is that Lola’s verbal comment is not really needed to give the 
viewer the impression of the metaleptical transgression. What the com- 
ment does is that it reminds the language-based original definition of 
metalepsis as the transgression of the frontier between “the world in which 
one tells, the world of which one tells” (Genette 1980 [1972]: 236, see 
even above). The fact that it is an intrusion into an ontologically different 
world appears more clearly in a movie, thanks to the visual dimension, but 
this does not mean that there are no narrative levels involved in this trans- 
gression. The world in which one tells corresponds to the world of the 
camera before it penetrates through the frame into the world of the paint- 
ing, after which the character of Lola can enter in direct dialogue with the 
viewer. To be more precise, this example corresponds to Genette’s first 
definition of metalepsis, “any intrusion by the extradiegetic narrator or 
narratee into the diegetic universe” (Genette 1980 [1972]: 234-235), 
since it is the narratee, meaning the recipient of the narration, who is 
entering the diegetic universe of the painting. Following John Pier’s 
dichotomy, this would be a descendant metalepsis, of the kind that Pier 
exemplifies with an extract from George Eliot’s Adam Bede (1859), when 
the narratee and the narrator together transgress the demarcation line 
between the narrative levels (Pier 2005: 250). 

However, there are even more interesting examples of metalepses in 
this movie, especially of the kind that could be analysed as horizontal (that 
is, I recall, metalepses which occur between worlds at the same narrative 
level). Indeed, after the metaleptical start, the film represents the events in 
the world of the painting in a traditional way, without several levels or 
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worlds involved, making the viewer forget the initial metaleptical intru- 
sion. All of a sudden, Lola, who has abandoned her ability to communi- 
cate with an extradiegetic narratee and has become a character inside the 
world of the painting, arrives at the frontier of her world, the canvas of the 
painting, and jumps out of it, followed immediately afterwards by two 
other characters. They enter an extradiegetic world, an ontologically dif- 
ferent world, the world of the creator, the painter, a world which could be 
the same as the initial world of the narratee, the viewers’ world. This is of 
course a hierarchically different world, so the transgression is undoubtedly 
vertical. According to John Pier’s typology, this would be the case of an 
ascending metalepsis, which he exemplifies with an extract from Mark 
Twain’s The Adventures of Huckleberry Finn (1883), more precisely when 
Huck goes up one level and enters Mark Twain’s world (Pier 2005: 252). 
This transgression is so important in the movie Le tableau that exactly the 
moment of the jump is used on the DVD cover. 

What happens next in the film, though, is very interesting from a theo- 
retical point of view. When Lola jumps into her creator’s world, she lands 
on the frame of another painting. Shortly after that, the second painting 
falls down from the wall because of Lola’s weight, and Lola falls through 
the canvas into the world of this second painting, representing a war. She 
becomes a character inside that world, able to interact with the sol- 
diers there. 

This example of metalepsis could be analysed as an ontological metalep- 
sis, as opposed to the more ‘rhetorical’ or ‘discursive’ cases studied above. 
Indeed, Lola literally travels between different worlds in a way that could 
be analysed through the lenses of the possible world theories. According 
to Alber’s and Bell’s attempt to apply these theories to metalepses (Alber 
and Bell 2012), such a travel between different ontological worlds would 
correspond, at least partly, to Daniel Lewis’s concretist philosophical posi- 
tion. According to the Concretists, as opposed to Abstractionists, such as 
Nicholas Rescher, Saul Kripke, Jaakko Hintikka or Alvin Plantinga, an 
individual cannot exist simultaneously in the same world. Thus, when Lola 
penetrates the world of the painter, she disappears from the world of the 
painting. 

What is more important is that this is a horizontal metalepsis, since Lola 
actually crosses the frontier between two worlds situated at the same level: 
the worlds of two paintings hanging on the same wall. It is this kind of 
metalepsis that Sonja Klimek finds incompatible with Genette’s original 
definition, since the transgression is between worlds, not between 
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narrative levels. But what this movie example shows is that in order to 
move from one painting to the other, Lola must necessarily pass through 
her creator’s world, the world ‘in which one tells’. 

One could argue that this example cannot be applied to literature, 
where the movement between two parallel worlds could occur without a 
necessary transition through a higher narrative level. But does the fact that 
such a transition is not represented imply that it does not occur? Is it not 
rather that the transition is implicit, and does not have to be spelled out 
explicitly? My answer would be that this depends. In some cases, such as 
when a character from a fictional work appears in a later fictional work, it 
is probably better to talk about what Marie-Laure Ryan calls “transfiction- 
ality”, meaning “the migration of fictional entities across different texts” 
(Ryan 2013). But in the cases when such a migration is actually repre- 
sented, when a character is represented when he or she literally leaves the 
storyworld, the paradoxical dimension is more obvious. However, even if 
I cannot provide a definitive answer as to the nature of horizontal meta- 
lepsis in the context of this article, it has appeared as obvious that a trans- 
medial perspective is necessary in order to better understand this 
narrative device. 

Before going over to the following film, I want to mention succinctly 
the metalepsis at the end of the film. What happens there is that Lola 
finally meets her creator, a painter, and talks to him, the way Augusto does 
with Unamuno. Her comment to him is, *But who painted you?" A ques- 
tion which is relevant even from a theoretical point of view since, indeed, 
the painter from the film is nothing else than represented, even though he 
is not an animated, but a live action character. The question also under- 
lines that metalepsis has the same effect in film as in literature, as we saw 
in Niebla: making one doubt about his or her own existence. 


6.3.2 Metalepsis Without Narrative Voice in Film 


The third film to be analysed is also the one in which the metalepses are 
created without any implication of a narrator. Thus, the language-based 
features are overshadowed by other features, both visual and aural ones. 
What is particularly interesting is that the aural features are not related to 
language, but to music, and the so-called world in which one tells has no 
relevance, since there is no narrator speaking. Therefore, the film is an 
illustrative argument against the language-based definitions of metalepsis, 
and a good example of the transmediality of the concept. 
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The film in question is an Icelandic-Ukrainian coproduction from 2018 
entitled Woman at War and directed by Benedikt Erlingsson. The plot is 
completely centred on the main character, Halla, a choir conductor and 
eco-activist, played by Halldóra Geirharðsdóttir, and on her attempts to 
sabotage an aluminum plant in the Icelandic highlands. From this point of 
view, the film would fit Jørgen Bruhn’s definition of ecomedia, that is, 
“representations of environmental issues in any specific media product” 
(Bruhn 2020: 121). However, at the same time, as Halla sabotages this 
plant in ingenious but dangerous ways, an application she made years ear- 
lier to adopt an orphan child from Ukraine is approved. The film follows 
the two different lines of plot in parallel, thus showing that illegal activism 
can very well be compatible with a humanistic act. 

Even if the plot is thrilling, the merits of the film lie rather in its cine- 
matic language, as critics have noticed. Just to mention one reaction, Peter 
Bradshaw, from The Guardian, finds the film “confidently and rather styl- 
ishly made” (Bradshaw 2018). However, Bradshaw finds this stylishness a 
bit exaggerated and unmotivated, especially when it comes to the use 
of music: 


There is also a lot of Icelandic folk music from a tuba, accordion and drums 
trio, and a rather beautiful singing group, whose performances turn out to 
be diegetic. That is, the camera pans around to show the musicians them- 
selves, standing weirdly, incongruously in the background of the shot, vari- 
ously trilling or parping away. It is a comic effect that is a bit distracting, 
subject to diminishing returns, and which ironises and undermines the 
action and obstructs your natural tendency to invest emotionally in Halla’s 
dilemma. (Bradshaw 2018) 


It is not my aim to contradict Bradshaw’s opinion, but as a specific fea- 
ture, the use of music is one of the most interesting details of Woman at 
War. Besides, it is through music that metalepsis is constructed in this 
film, and that is done in innovative ways. This is a good example of what 
Werner Wolf in an article calls ‘musical metalepsis’, which he contrasts 
with verbal or pictorial metalepses, that is, metalepses constructed through 
verbal or pictorial means (Wolf 2019). As it appears clearly in Wolf’s arti- 
cle, instrumental music alone, as a media product, cannot achieve meta- 
lepses. It does so only when it is incorporated in film: “what in ‘absolute’ 
instrumental music would be impossible but which the plurimedial 
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combination of music, narrative and the moving image in the sound film 
can produce, namely musical metalepsis” (Wolf 2019: 29). 

I follow Wolf’s definition of the concept of musical metalepsis, and 
analyse how music can be used in order to create metalepsis in other media 
where it is used, since Woman at War is particularly interesting from this 
point of view. Indeed, Bradshaw is right to point out that what is impor- 
tant in the use of music in this film is the performance of the musicians. 
Admittedly, the music itself is important in the film, from different points 
of views. Already the fact that Halla is a choir-leader, with several scenes 
presenting her while leading the choir, is a sign that music should be given 
particular attention. The score itself, specially composed for the film by 
Davíð Pór Jónsson, is original in its minimalism. Generally, it seems to be 
used in conventional ways, meaning the ways it has been established in 
classical Hollywood film. Here is a short list of the possible functions of 
music in such films: 


It can establish setting, [...] it can fashion a mood and create atmosphere; it 
can call attention to elements onscreen or offscreen, thus clarifying matters 
of plot and narrative progression; it can reinforce or foreshadow narrative 
developments and contribute to the way we respond to them; it can eluci- 
date characters? motivations and help us to know what they are thinking; it 
can contribute to the creation of emotions [...] [it] can unify a series of 
images [...] [and] encourages our absorption into the film by distracting us 
from its technological basis [...]. (Kalinak 2010: 1) 


All these functions apply to the use of music in Woman at War. 
However, there is much more to it. To start with, still in the traditional 
way, music highlights the double action of the film. The Icelandic music 
trio, who play the drums, the piano and the accordion, generally play 
music that reinforces the atmosphere of danger that has to do with Halla's 
sabotaging of the aluminium plant. The other group, a Ukrainian female 
choir singing folk songs, is used for the other plot line, that is, the adop- 
tion of the Ukrainian orphan. The fact that the two groups only play 
together at the end, when Halla meets the girl she adopts, could be seen 
as a metaphorical use of music, to “clarify” the plot, as Kathryn Kalinak 
mentions in the quotation above. 

As Bradshaw notices, music is also used in unexpected ways in Woman 
at War. One of these ways enters indeed in complete contradiction with 
what Kalinak listed as a function of film music, namely that it is 
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“distracting us from its technological basis”. It is exactly when music is 
used metaleptically that this happens. Indeed, already in the first scene, 
when the credits still run, the music suddenly seems to lose its extradi- 
egetic status, since the Icelandic trio appears in the frame, when the cam- 
era pans the beautiful Icelandic landscape. They stay there, “weirdly, 
incongruously” in the field, as Bradshaw rightfully notices. What Bradshaw 
misinterprets, however, is that music thus turns out to be diegetic. Indeed, 
just because the musicians are visually represented does not mean that they 
become characters in the diegesis. It is a misconception, only caused by 
the conventions from traditional films, that everything that is seen on the 
screen exists at the same level. It is enough to reconsider this by applying 
the mode of hearing, which should apply in the same way: everything that 
is heard in the film exists at the same level. Which obviously is not true, 
since extradiegetic music, or extradiegetic narrators in form of voice-over, 
are used frequently. So why could not extradiegetic elements be visually 
represented on the screen without “distracting? the viewers, as 
Bradshaw claims? 

There are, indeed, clear arguments against considering the music in the 
film as diegetic. Actually, the music is not heard by the characters. Only 
two characters hear the music the same way as Harold Crick hears the nar- 
rator’s voice-over: Halla (a couple of times) and a Spanish backpacker who 
happens to be close by when Halla is hunted in the highlands (once). 
When these two characters show that they hear the extradiegetic music, it 
is when metalepsis actually occurs. Because, I argue, until the two differ- 
ent levels, the extradiegetic and the diegetic, interact, there is no real 
transgression. Or rather, I should say, the transgression is covered the 
same way as the use of extradiegetic music is covered by its frequent use. 
Because, as I already mentioned above in the analysis of Stanger than 
Fiction, all coexistence of diegetic and extradiegetic elements is transgres- 
sive, and thus metaleptical. So, what these examples show is that metalep- 
sis actually loses its raison d’étre when it is used in conventional ways. 

The extradiegetic and the diegetic collide in another way too in Woman 
at War. This is also through music, or rather through the visual represen- 
tation of the performance of the musicians. Indeed, the Icelandic trio 
actually interferes with the events in the diegesis at least three times. Two 
of these instants involve the use of music. Indeed, by changing the rhythm, 
the musicians warn the two characters that are able to hear them (Halla 
and the Spanish backpacker) that something dangerous is going to hap- 
pen. In both cases, the characters react, and the course of action is changed: 
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rather slightly in the case of the Spanish backpacker, who turns his head 
and discovers Halla running in the fields, and more importantly in the case 
of Halla, who realizes that the police are stopping people at the airport, 
and decides not to take that flight. The third case is even more clearly 
metaleptical, since the musicians physically interfere with the events in the 
plot. It is when, while playing the extradiegetic music as usual in Halla’s 
apartment, they decide to put on the TV set so that she can find out that 
the police have started the hunt. The TV set thus functions metaleptically 
as well, since it is at the same time both a diegetic screen and a metadi- 
egetic screen, according to the distinction made by Andrea Virginas in this 
volume (2020). 

In conclusion, the musical metalepses in Woman at War are particularly 
interesting to study in a transmedial perspective, and by concentrating on 
the modalities of the media product, according to Ellestróm's model of 
intermedial relations (2020). Indeed, what appears clearly is that when the 
mode of seeing is applied to extradiegetic music, that is, when the musi- 
cians are visually represented, what was hitherto a conventional use of 
extradiegetic music can be experienced as metaleptical. More importantly, 
when the effects that the extradiegetic world has on the events in the dieg- 
esis are visually shown, the metalepsis becomes clearer for the viewer. 


6.4 CONCLUSION 


This chapter did not aim at changing the definition of metalepsis. Its first 
definition, formulated by Genette in 1972, is still valid as a general starting 
point: “any intrusion by the extradiegetic narrator or narratee into the 
diegetic universe (or by the diegetic characters into a metadiegetic uni- 
verse, etc.), or the inverse” (Genette 1980 [1972]: 234-235). I tried to 
show that metalepsis is a transmedial device, which can appear even in 
media products that are not language-based. What seems to be mandatory 
is the capacity to represent. And I thereby do not mean to represent in 
general, because all media represent in some ways, but to represent in 
some detail such features that are necessary for perceiving, for instance, 
ontological clashes and narrative levels. 

What I particularly wanted to show is that the concept of narrator can 
obstruct the expansion of the applicability of metalepsis to other media. 
An intrusion, or rather a transgression, is needed, between two worlds, 
but these worlds do not necessarily have to be created by a narrator's 
voice. Consequently, the initial definition can be improved, but it already 
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possesses a flexibility, since it mentions “diegetic universes”. Admittedly, it 
lacks even a specific reference to horizontal transgressions, that is, between 
parallel universes, but I argue that that kind of transgression can be stud- 
ied within the same framework, since the parallel universes are necessarily 
created from a superior level. 

In order to prove my point, I started with the media type in which 
metalepsis was first defined, that is, literature, and studied one of the best 
known examples of metalepsis: the meeting between Unamuno, the 
author and the narrator of the novel Niebla, and a character created by 
him, Augusto Pérez. A closer look showed that even if the narrator is 
important in that scene, the most shocking is the representation of the 
whole meeting. Indeed, it is first when Unamuno, the narrator, becomes 
a character who physically meets Augusto that the metalepsis becomes 
really obvious and astonishing, both for Augusto (and eventually even for 
Unamuno) and for us, readers. What is interesting is that the shocking 
effect can be expanded to existence in general, since ultimately, even in 
reality, human beings can have the impression of being the creation of a 
God. This effect is obviously possible even in the case of other media 
types, as is explicitly mentioned in the film Le tableau. 

After the literary example, I analysed metalepses in three films, chosen 
not only because they actually contain metalepses, but also because they 
create it in different ways. The first example, Stranger than Fiction, is also 
the one which is the closest to metalepsis in literature, since the metalepses 
are occurring between a narrator, whose voice imitates a narrative voice in 
literature, and a character created by this narrator. However, the ways in 
which the metalepses are achieved rely especially on the mode of sound, 
which is absent in literature. So, even if what we have in this film is a so- 
called intermedial imitation of a narrator, the ways in which that is done 
highlights the transmedial potential of metalepsis. A potential that is even 
clearer in the other two films that I analyse, since they take gradually 
greater distance from the use of a narrator. 

The second film that I analysed, Le tableau, starts by representing a nar- 
rator who performs a metaleptical transgression, but this initial imitation 
of a literary narrator is not repeated, and the metalepses later on are con- 
structed with the help of filmic media characteristics. One of these meta- 
lepses, constructed by applying to the visual mode, can be used as an 
argument for the possibility of horizontal metalepses. 

The last film, Woman at War, is the one in which no reference is made 
to literature when it comes to its metalepses. Here too, it is through the 
mode of sound that the metalepses are principally created, but this time by 
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using the qualified medium of music. What the analysis shows is that these 
‘musical metalepses’ are all the more effective when the visual mode is 
used too, showing music as a performance act. 

All in all, the analyses have shown that a transmedial perspective on 
metalepsis is beneficial, even in the case of literature. This is in line with 
Ellestróm's view on media not as essential and hermetically closed entities, 
but as open and flexible concepts (2014, 2020). It has also proved pro- 
ductive to concentrate on the modalities of media, and to analyse how 
different modes can affect metalepses. What the analyses have shown too 
is that the complex device of metalepsis can be used to illustrate aspects of 
Ellestróm's model of intermedial relations, especially those related to 
modalities (2020). 
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71 INTRODUCTION: BEING IN THE WORLD 


All humans have experiences from being in an environment. For most of 
us, this is based on movements within some sort of space,' and the experi- 
ences can include cycling in the woods, walking a city street, or riding 
through mountains. Exceedingly, over the last two centuries, this move- 
ment has been supported by engine driven devices, such as cars, trains, and 
airplanes; in earlier days, oars, wind, and animal energy were used for 
transport. The experience of existing in the world involves interaction 
with a landscape where other beings also dwell and machines are used. An 
environment where humans and other animals coexist can pose risks, sup- 
port, and cooperation amongst other types of impact for the individual. It 
provides, in the sense established by Gibson (1986), affordances. The 
affordances go in different directions, forming networks of connections 
between beings based on a rich set of possible forms of communication 
(Ingold 2000). 

One key element of communication is the landscape itself. Many funda- 
mental aspects of communication, both in general and of human lan- 
guages specifically, are connected to spatial organisation and movement 
(Levinson 2003; Tversky 2019). This is the case for all types of media used 
in human communication. Spatiotemporal modes are active in all media 
products, however different they may be in different types of expressions. 
The decoding and understanding of these modes as part of the spatiotem- 
poral modality of a media product is always, in one way or another, con- 
nected to the recipient's embodied memories of experiences in an 
environment. 

Such memories can be specifically linked to a concrete reference. In a 
novel, where the events take place in Prague, the reference to Náméstí 
Republiky is recognisable as a concrete place by a reader who knows the 
place. References can also be type based. The island in which Robinson 
Crusoe was shipwrecked cannot be experienced in the same way as the 
square in Prague because it does not exist. However, the landscape 
described is still recognisable by readers based on general knowledge of 
landscapes, including beaches and forests. The embodiment is also linked 
to the situatedness of the reading process, as pointed out in Have and 
Pedersen (2020: 200). 

Space, communication about space, and media, including transforma- 
tions between them, are all interwoven. In this chapter, we will focus on 
expressions describing landscapes in different media. In order to better 
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understand how different media are used to communicate about and con- 
vey understanding of landscapes, one central aim of this chapter is to 
include a focus on the transformative processes connected to the expres- 
sion of spatial understanding. We begin by discussing traditional ways of 
expressing knowledge about landscapes: written and oral texts and maps, 
and their combination in geocommunication systems. Furthermore, we 
will use the four media modalities explained in Ellestróm (2020) as a sys- 
tem for understanding and comparing these media forms as they are used 
for spatial expressions. By focusing on transformation processes, we will 
aim to attain a better understanding of the limitations and possibilities 
within this approach. Moving beyond graphical two-dimensional expres- 
sions and written texts describing landscapes into virtual three-dimensional 
‘landscapes’ will enable us to better comprehend the spatial media charac- 
teristics influencing media products. 

The relationship between the spaces in a virtual world and the spaces 
we know as embodied animals is complex and many-faceted. This estab- 
lishes virtual reality (hereafter: VR) both as mediated expressions (VR is a 
type of communication device) and as an experienced space, linked to an 
equally complex set of connections between different forms of embodi- 
ment and different sets of affordances. 

These claims will be further explored through discussing landscapes in 
a number of different senses. We will start off with the real environment 
we move around in as embodied creatures. We will focus on how this has 
been mediated through centuries and millennia of human culture. We will 
then move over to the landscapes we move around in through avatars, that 
is, VR. That enables us to discuss in a more abstract way connections to 
modelling and modelling processes in general. This will include studying 
concrete types of spatial expressions through new media, such as VR appli- 
cations. These are less affected by convention and have a much shorter 
tradition of developing their ways of using the available affordances, due 
to being new media in a historical sense. The technical development also 
changes the affordances themselves. 


7:2, DESCRIPTIONS OF LANDSCAPES 


The human mind-body system has the capability of linking media prod- 
ucts, expressed in a wide range of different media, to embodied experi- 
ences. This is used when creating media products based on pre-knowledge 
on the side of the creator, from oral, written, and pictorial media and 
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dance to films, computer games, and VR. We will begin this chapter by 
focusing on traditional ways of making descriptions: texts, written and 
oral, and maps.” Furthermore, with two-dimensional static documents as 
our starting point, we will use the media modalities as a system for under- 
standing these specific media forms in a structured way. This will show us 
that they are similar in both material and sensorial modalities, as they are 
both flat documents approached mainly through seeing.* The differences 
between documents carried by computer screens versus paper and similar 
technical interfaces will not be discussed at this stage. 

One main connector between texts and maps is place names, which are 
used by both media types to reference something external to the text and 
map. The links from documents to this externality can have different levels 
of realism. The reference function of place names is fairly similar in text 
and maps, but their documentary contexts are fundamentally different. 
This links to the spatiotemporal as well as the semiotic modalities, as we 
will see below. 


7.2.1 Written Texts 


Textual documents using the writing systems we know from languages 
such as English are generally based on a basic level of sequentiality, where 
letter follows letter and word follows word. This is the case even when 
writing directions vary (for instance, left-right, right-left, top-down) and 
some writing systems have a less strict sequentiality. The sequentiality is 
expressed in a spatial form and it is never absolute—there is always a pos- 
sibility of moving back and forth in texts. Some forms of writing, such as 
certain poetic forms and footnotes, are designed to break the sequential 
pattern of reading; however, this becomes an addition to the low-level 
sequentiality. The order of the letters making up a word and the words 
making up a sentence are fundamental for the semiotic systems of symbols 
that languages constitute even if this order can be broken and played with. 
The sequential conventions are elementary parts of the code of writing 
systems. Still, these sometimes differ and are not as ‘absolute’ as the space 
of a figure, such as a map. 

In this sequential structure, a place name is a word (expressed as a num- 
ber of characters) which is related to other words around it (Firth 1957: 
11). Syntactically, the primary relationships are within the sentence. When 
the same page of text includes several place names, the relationships 
between the targets of their denotations are not based on the actual 
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locations on the page, but rather on the meaning which can be found in 
the text. To competent readers, the locations described are decodable 
through the relationships expressed within the text, in context of their 
own pre-knowledge. The space of the landscape referred to by the text is 
a reconstructed space and the semiotic system used in the text, also to 
relate to the landscape, mostly relies on symbols. The reconstruction of 
the referenced space happens when the reader interprets the meaning of 
these symbols, and understands the links between strings in the text and 
places in the landscape. This reconstruction is not dependent on whether 
the landscape is understood as a real place by the reader, but rather on 
whether it is recognisable or imaginable. This is partly based on the 
embodied experiences discussed above, partly on impressions from media 
products, and partly on other inputs. This implies that the distinction 
between VR and other media is not essential for our ability to recon- 
struct spaces. 

The spatial relationship between places can be established in two differ- 
ent ways. It can be expressed directly by relational language constructions 
such as ‘north of’ or ‘down the river.” In this case, landscape descriptions 
can be understood even if they are not connected to any landscape pre- 
known to the reader. The language constructions establishing relation- 
ships between places are sufficient to get a basic understanding of the 
landscape described, but they are usually strongly under-specified (Eide 
2015: 120-125). 

Spatial relationships can also be assumed based on the knowledge ofthe 
specific landscape being described. In this case, the mere mentioning of 
two place names will establish a spatial relationship between them if they 
are both known to the reader, as long as ‘known’ implies that the locations 
of both places are known. No other language mechanisms than the place 
names are necessary in order to express such relationships. In neither case 
is it necessary that the referred places are real. The relationships between 
places in J.R.R. Tolkien’s Middle Earth are known to many readers, even 
if there is no physical landscape to reference. However, in the case of ficti- 
tious worlds, the two places in the last example must belong to the same 
‘world’. 

In usual reading, these two ways of establishing relationships are uti- 
lised together. This includes relational expressions such as ‘three miles 
down the river from Prague’ or ‘the hills between Prague and Wrocław.’ 
References to places can be expressed in a number of different ways in dif- 
ferent languages, and are often mixed with other references, to places as 
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well as to other things. In German news, the capital is often used to refer 
to a country. ‘Moscow claims that ...’ is a typical reference to the govern- 
ment of Russia, where the capital acts as a reference to the city of Moscow, 
where the government is located. “The Municipality of Oslo’ can refer to 
a geographical area, an actor, a community, a group of people, and more. 
Another variable factor of establishing these relationships is the distinc- 
tions between different languages used. Indo-European languages have 
well-developed preposition systems for expressing spatial relationships, 
whereas Finno-Ugric languages usually rely on grammatical cases. 


7.2.2 Map Documents 


Similarly to the aforementioned reference system for written text, the way 
we reference places on maps includes the use of place names, which can 
establish links from the map to a location through pre-knowledge in the 
mind of the reader.? However, the only understanding needed in order to 
read the precise spatial relationship between two places claimed by the 
map is to grasp the geometrical reference system of the space of the map. 
Once the scale and direction of the map is understood, each pair of places 
can be precisely related to each other by measuring the distance and the 
direction between them on the map. Furthermore, unnamed places are 
related to all other places in the same way. A black dot on the map can, 
given a certain map schema, be understood as a boulder, without any 
name needed in order to express the claim that this is a specific boulder in 
the external landscape referred to by the map (MacEachren 2004). 

The links are not just denotational, they are also connotational. The 
black dot on the map also links to the concept of what a boulder is (given 
it is known to the map reader), in addition to linking to the reader’s 
knowledge of a specific boulder in a real or fictitious landscape. Place 
names on maps are not needed in order to refer to particular places, but 
they help the map reader establish the connections to specific places. A red 
dot can be understood as a town, and letters next to the red dot will usu- 
ally establish the claim that this town has the name expressed by the letters. 

Nothing can be put on a map without simultaneously making claims 
about how the mark on the map relates to other marks on the same map. 
Relationships can be fuzzy, but they are there; borders of the fuzzy area 
indicated can be read from the map, establishing a span of the possible 
direction and distance between two places based on the scale and orienta- 
tion of the map.^ Even unmarked places on the map (the white areas) are 
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related spatially to other parts of the map, in their expressing ‘nothing 
relevant enough to be put on the map exist in this part of the space referred 
to by the map.’ An interesting difference between maps as they are dis- 
cussed here and architecture is pointed out by Miriam Vieira: maps 
describe /model space, whereas the role of architecture is to create space 
in the sense of altering or organising it (Vieira 2020: 60). 


7.2.3 Oral Texts 


We will now widen the scope beyond texts and maps expressed on flat 
documents, and proceed to discuss spoken words. Up until the spread of 
recording equipment in the first half of the twentieth century, spoken 
words were linked to presence in time—the message could only be heard 
at the time it was uttered.” The message was created, transferred, and 
received within milliseconds. Additional understandings of the message 
could follow much later, in the same way as when one only understands 
the full meaning of a warning when what is warned against actually takes 
place. In other cases, the creation of the message could be a reproduction 
of an expression that had existed for a long time, as in the re-performance 
of oral tradition. 

While oral communication often included visual presence, this was not 
necessary in the same way as time based presence: it could be dark, foggy, 
or there could be physical hindrances between the communication parties 
(as in the case of catholic confession). Since the arrival of recording equip- 
ment, the time presence is no longer obligatory for the oral communica- 
tion to carry through. We now have the capabilities to communicate not 
only over long distances (as with TV and radio) but even across time. This 
secondary orality (Ong 2002) is relevant to communication about space 
due to the disappearance of the present human body from the communi- 
cation situation. Until around 1900, only the human body in itself was 
available as a combined technical medium for and producer of oral text. 
This meant that gestures and facial expressions were usually available in 
most communication situations, as was the possibility to comment and ask 
questions. This is no longer the case, which has led to new forms of oral 
communication—new oral qualified media types (cf. Ellestróm 2020: 
54—60). We will now move into a more detailed analysis of the conse- 
quences and influences of this for oral communication about space. 

One example of oral communication about space is the explanation of 
a travel route from one person to another, discussed by Vieira (2020: 71) 
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as a mental tour. This interaction involves a series of place references 
within a real landscape, in the form of place names either in a strict sense 
or as other spoken words and/or non-verbal sounds. Oral communication 
is based on the technical interface of sound, where hearing is the sense of 
reception. This means that there is time manifested in the material inter- 
face, but no space—almost. There are cases where the sounds of the pro- 
nunciation of words represent space in an onomatopoeic sense (‘the road 
is straight for a lo-o-o-ong time’). The musical aspects of chanting about 
places, where, for instance, heavily marked high-pitched tones can be used 
to represent mountain peaks, can describe space through auditory iconic- 
ity (Tirén 1942: 47; cf. Graff 2004). However, in the material we have 
worked on, these examples are not frequent, leaving symbolic representa- 
tions establishing virtual spaces as the main way to represent spatiality 
through the sound of oral communication. Ellestróm (2020: 78) points 
out that, in the case of pop music, there is a mix between auditory text and 
auditory image. This mix is always there in oral texts, but the balance 
between the aspects varies. 

If we consider the situation of face-to-face oral communication, the 
parties have access to a number of additional affordances. One can ask 
questions and request clarification, but one can also use gestures and other 
bodily movements to indicate spatial aspects, even in an indexical sense: 
‘We went to that mountain [gesturing the direction towards a visible 
mountain peak].’ Furthermore, oral spatial communication is a common 
way of using maps (Hyttfors and Tirén 2011: 5), as the basis for a series of 
gestures indicating past or future travels. These gestures may also include 
leaving traces with a pen or a pencil (Wood 1993). In this sense, it can 
include traces of human bodily acts in the material interface. 

Oral maps do exist as a concept, but they are quite insignificant com- 
pared to texts and map documents. They make up neither a technical nor 
a qualified medium. The concept is used mostly for descriptive oral texts. 
As opposed to maps as they were defined above, they do not have space 
manifested in the technical interface and they are approached mostly 
through hearing, even if some gesture can be involved when the commu- 
nication happens in real time between people present at the same place 
and seeing each other. What is called oral maps represents a subclass of 
(and is therefore classified with) oral texts, with an especially strong focus 
on space. 
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7.3 GEOCOMMUNICATION 


All the media described above can be referred to as ‘maps’ in ordinary 
language, and this use of language also stretches far into academic com- 
munication in several disciplines (Eide 2021). Ellestróm (2020: 56) claims 
that everyday language only covers a few basic media types. This is true. It 
is also the case that the media types that are covered are not based on clear 
definitions ofthe terms. Furthermore, as pointed out by Simonson (2020), 
media borders are constantly being pushed and will change over time, due 
to the active engagement with and across the borders of media, in line 
with Ellestróm's description of qualified media (2020: 54-66). 

Additionally, the word *map' is extended further to include, for instance, 
mental construction in human minds (cognitive maps).? While this use of 
language is common also historically, it is inconvenient in scholarly work. 
Hence the more limited definition of ‘map’ given above. It is nevertheless 
useful to have a wider concept for the media-based methods humans and 
other animals use to communicate about landscapes. In this chapter, we 
will follow Brodersen (2008) in using the word ‘geocommunication’ for 
this purpose.? 

All the media forms described above fall into the broader concept and 
can be described as geocommunication. That also includes other forms 
such as braille text and maps, smell, sign language, and dance. These other 
forms, common among a number of animal species, will not be discussed 
any further in this chapter. Vieira (2020) shows how an equally complex 
space/time situation is found also in architecture, linked to the negotia- 
tions between the persons involved in practical work. All situations where 
the qualified media listed above are used also have a potential for including 
other aspects of geocommunication. 


7.4 REFERENCES BETWEEN Maps AND TEXTS 


When studying different media and media products, it is important to also 
focus on the links between them. The reference functions of maps and 
texts describing space do not only link media products to landscapes, real 
or fictional. Additionally, they link media products, internally and with 
other media products, through the existence of co-reference (Eide 2009). 
Co-reference is a basic feature of human communication. In linguistics, it 
is used in the resolution of anaphors: *We went to Rome. We had a good 
time there.’ In this example, ‘Rome’ and ‘there’ co-refer; they both have 
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the city of Rome as their denotation. Co-reference in text is fundamental 
in the establishment of spatial meaning, way beyond anaphor resolution. 
Most texts include a series of references to locations, in which co-reference 
is used to establish connections between such references. In editions of 
texts, such as historical source material, it is common to include a gazet- 
teer, a list of place names mentioned in the text with references to where 
they occur in the text as well as information about their assumed real or 
fictitious location. The place name occurrences grouped under one head- 
word in a gazetteer are grouping together a set of co-referring place 
names.'? This can also be done when the denotation is uncertain or not 
known at all, or when it is assumed to be a non-existing place. The crite- 
rion is common denotation, not the nature of what is denoted. 

This form for co-reference does not exist in any single map image, as 
one place in the referred landscape can only be represented as one visual 
item on the map image. Only in multi-image maps, such as printed maps 
with a zoom function or with a series of maps depicting the same area, 
does the understanding of the map include a reliance on co-reference. The 
co-references link locations on the main map image and the zoomed map 
image, or in multiple map images denoting different aspects of the same 
area. Co-reference is also used in rectification of digitised maps, where 
points on the scanned map are linked to points on a digitally georefer- 
enced map. The scanned map is morphed to fit the digitally georeferenced 
map and is thus fixated to a standard projection. The rationale for con- 
necting two points in rectification is that they refer to the same place in the 
space referred to by the two maps. 

Connections between documents in the form of texts and maps are 
established by a competent reader in a comparable way. When using a map 
to trace the movements expressed in a travel narrative, places in the text 
are linked to places on the map through denotational co-reference. Place 
names in both are important; however, identity in names is neither a nec- 
essary nor a sufficient condition for co-reference. Relational expressions, 
as discussed above, can be used. Furthermore, the names have to be inter- 
preted correctly. Someone reading a travel narrative from Canada may 
know that there is a much larger ‘London’ in England, but may still choose 
London, Ontario, as the place on the map she assumes ‘London’ in the 
text co-refers with. On the other hand, reading about ‘Königsberg’ in a 
book on the history of mathematics, it is of no avail to look for the identi- 
cal name on most modern maps; Kaliningrad or Kanmuunrpag will be the 
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two most likely series of letters to be found referring to the same city as the 
one the historical name ‘Königsberg’ refers to. 

Automatic mapping of texts, which is more popular than ever in the age 
of widespread digital maps, is done in a similar way. A computer pro- 
gramme is used to identify all place references, usually focusing on place 
names. Then these names are run through another programme to geo- 
refer them, that is, to find geographical coordinates for each of the place 
names. Various types of cleaning processes may be added in order to find 
the most likely candidate for several different places with the same name. 
These can include both digital decision-making systems and human inter- 
vention. The next step is using the coordinates to position the place names 
on the map, possibly with links to frequency information, the context 
around their occurrences in the text, and other relevant information. The 
quality of such systems varies depending on the nature of the source texts, 
and for historical sources, it is hard to reach acceptable quality without 
human intervention. 

So where do we find co-reference here? The assumption in the creation 
and use of such tools is that the geocoding can identify a place with coor- 
dinates based on occurrences of names of the place, so that the coordi- 
nates refer to the same place in an external landscape as the names in the 
text. Ideally, this process is supported by additional criteria and can iden- 
tify places across different names, while distinguishing between different 
places with the same name. Without the co-reference assumption, the pro- 
cess would not make any sense. 

Mapping of written texts is a much more common procedure than 
mapping of oral texts. When working with recorded oral material, such as 
historical sources, the normal procedure would be to first transcribe the 
oral recording into a written text, and then identify and map the place 
names in the transcript. There are cases, however, where oral storytelling 
is mapped directly. When other phenomena than the mere existence of a 
place reference in the oral text is to be mapped, an interactive situation is 
needed in order to capture the richness of the knowledge possessed by the 
persons being interviewed. The mapping also includes additional qualita- 
tive aspects. In the tradition of mapping indigenous historical use and 
occupation of land based on oral testimony, gestures and other embodied 
movements complement oral texts in the telling of historical spatial sto- 
ries, which are then transferred to a static paper map, a so-called use-and- 
occupancy-map (Tobias 2009). 
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These maps document indigenous use of land, based on interviews with 
witnesses about their memories. The stories told come into being in inter- 
action between the witness and the interviewer. In this interactive perfor- 
mance, time is manifested in the material interface. Parts of the information 
exchanged in the interaction are transferred to the map where it is recorded 
as a timeless trace—it is moved from time to space. Gestures make up an 
important part of the performance, and some of these lead to traces on the 
map. While being expressed on a ‘timeless’ document, these traces can 
include cues which can be used to re-establish some of the time aspects 
from the interviews. Combined with other pieces of evidence, the docu- 
ment is used to establish a coherent understanding of the often overlap- 
ping historical use of the land. This is then used as documentation in 
future work on land rights issues, often including legal processes. 

Creating texts based on maps is not common. Classical ekphrasis, where 
poetry describes works of plastic art, is not common in texts with maps as 
sources, even if it happens in architecture (Vieira 2020). Map-based texts, 
however, are well known from other situations. In fiction literature, espe- 
cially in novels or series of novels where the site of action is a fantasy world, 
it is common to draw maps of the spaces before the text is written.!! While 
the text in those cases is not a description of the map, they are still based 
on the understanding of a fictitious landscape as it is expressed on the 
map. The map works as a geometric structure for the text—as a figure 
used to organise the spatial aspects of the meaning to be expressed in 
the text. 

Textual descriptions of maps are found in scholarly works, for instance 
on the history of cartography.” Facsimiles of maps discussed in detail tend 
to be included. In these cases, the description is a guide for the eyes of the 
beholder, rather than a text meant to replace the map. Additionally, in 
such cases, the denotational function of the map rarely plays a significant 
role in the textual description. 


7.5 ON THE REALISM OF LANDSCAPES 


Until now, we have described landscapes referred to from texts or maps as 
either real or unreal. This is a simplification of how texts and maps work. 
While there are clear rhetorical cases of claims for existing landscapes, such 
as in historical texts about The Second World War, and claims for non- 
existing landscapes, such as Narnia, these are only two specific ways of 
referring media products to landscapes. In works of fiction, the landscape 
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described in the text is often an adjusted version of the real world. For 
instance, in Robinson Crusoe, the world described is the one we know, 
with the exception of the island where Crusoe spent his 21 years after one 
of the shipwrecks in the book. Map examples include Soviet propaganda 
maps from the cold war where places were systematically displaced 
(Monmonier 2018: 125-129). 

Any linking of a place in fiction to any map is part of an interpretative 
act, including the assumption that the fictitious text can meaningfully be 
mapped onto a real world map. But what is a fictitious text and what is 
not? For some people, texts describing the Apollo 11 expedition are ficti- 
tious, and the truth/fiction nature of religious texts is permanently con- 
tested. While the fact that the links to real world maps is an act of 
interpretation is important in scholarly projects creating maps of literary 
works, it is questionable how relevant it is for a general reader. Reading is 
organised according to the principle of minimal departure (Ryan 1980) in 
the understanding expressed by Ellestróm (2018: 27): the extra- 
communicational domain, which is the prior knowledge of the reader, is 
key to establish an understanding of any work of art. As pointed out 
already by Ryan (1980), deviations from prior knowledge are only made 
when unavoidable. 

When an assumed non-fiction text is read as a presentation of actual 
history or geography, the principle of minimal departure is also applied. 
Deviations from what the reader knows from beforehand will be negoti- 
ated. However, whether the results of the negotiations include changes to 
the world view of the reader in her extra-communicational domain will be 
influenced by the reader’s assumptions as to whether the text is fictitious 
or not. If the work is assumed to be fictitious, the reader will create an 
adjusted world view for that work of fiction and others based on it (sequels, 
adaptations, fan fiction, etc.). The assumed world of Robinson Crusoe 
includes the island where he shipwrecked, but the assumed real world does 
not, even after reading the book. If the text is assumed to be non-fiction, 
the change to the world view stands a much higher chance to be general 
to the world view of the reader, such as a new atlas correcting the locations 
and forms of the misplaced Soviet places from the cold war. This is a pro- 
totypical distinction which glosses over many cases not clearly located in 
either category. Historical fiction is often assumed to have a true historical 
core, so that a reader might adjust her general extra-communicational 
domain for at least some of the adjustments she makes during reading. As 
the level of persuasiveness is not directly linked to the validity of the 
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underlying claims, as pointed out by Tseng (2020), the adjustments can 
only be understood in relationship to the openness for the persuasions 
from the reader’s perspective. 

Further complexities are added by the fact that maps, being models, 
always simplify what is mapped. There are a number of systematic devi- 
ances on maps. One example is a traditional paper based 1:50,000 scale 
map depicting a road and a house. The road is drawn as a black line, the 
house as a black square, and the real world distance between the middle of 
the road and the middle of the house is 20 metres. Each map symbol exag- 
gerates the size to avoid them being too small to be identified. The road 
might be depicted as 1 mm wide, and the house a rectangle of 1 times 
2 mm. Given the scale of the map, 1 mm represents 50 metres. Thus, if the 
location of the two symbols were precise they would overlap and, being 
the same colour, the house would normally be read as a small parking lot 
or some other part of the road. In such cases, not only the size of the sym- 
bols but also the locations are adjusted. The house is typically moved at 
least 1 mm away from the road, possibly more. Based on the fact that this 
is the smallest distance between two such symbols used on this series of 
maps, the house will be read by a competent map reader as being next to 
the road, without any other assumption about distance than it being less 
than around 100 metres. 

All these types of situations, of which many more exist, show that nei- 
ther maps nor texts are either true or false, and understanding the com- 
plexities in how they relate to facts and fiction is dependent on the readers 
competence. These complexities similarly exist in new media, such as com- 
puter games and VR. What sets these apart is the way the environment is 
experienced in new media, which creates different situations, as we will see 
below. A competent reader will have well-founded assumptions about the 
realism of texts and maps, which will often work well. However, in some 
cases, they will be imprecise or even downright wrong, as when a fiction 
book is read as if it were non-fiction. Going forward, we will therefore not 
assume that fictitious and real maps are two different categories in real 
media use. We will instead reason in terms of degrees of realism in the way 
a landscape is depicted by maps and texts. Realism must here be taken 
both as intention by the author or cartographer and as testable empiri- 
cal facts. 
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7.6 Mars Arg Nor Texts, But NEITHER ARE TEXTS 


When reading a written text, listening to an oral text, or reading a map, 
one interacts with two different types of material interfaces using two dif- 
ferent senses. The written text and the map are both flat documents with 
space manifested in the material interface, whereas the oral text is made up 
by vibrations in the air with time manifested in the material interface. The 
latter is often complemented by a human body using arms, fingers, and 
other body parts, as well as external objects, to indicate directions and 
make references. These three are also often combined, in either pairs or all 
three together. This can also be the case with other media types such as 
VR, as we will see below. Humans possess a wide area of mixed geocom- 
munication methods. In this section, however, we will see each of these 
analytically isolated as a specific media type with its specific modalities. 

What our eyes see and what our ears hear lead to meaning. This transfer 
process has been studied extensively over millennia. In the following 
description, the basis is Ellestróm's four media modalities (2020). These 
modalities do not indicate any essential division (the material interface has 
no clear border to the spatiotemporal modality); the distinctions are ana- 
lytical. Furthermore, there is no obligatory development in time from one 
modality to the other. We do not first meet a material interface, then use 
our senses, then go through spatiotemporality so that we, in the end, can 
apply semiotics. The different parts of the process are all mixed together. 

As one may expect for media with a special link to landscapes: the spa- 
tiotemporal modality offers a certain amount of complexity when we study 
maps, written texts, and oral text (Ellestróm 2018: 19). What is estab- 
lished as senses interacting with physical material (in our cases sight and 
documents or hearing and sound waves in the air) goes through the spa- 
tiotemporal tunnel into a world of meaning. Here the signs seen and heard 
make sense in interaction with the pre-knowledge of the specific 
human mind. 

In the sensorial and material modalities, the two types of spatial docu- 
ments—written texts and maps—are similar to each other, but quite dif- 
ferent from oral text, which is one separate track of sounds in the air. In 
the semiotic modality, on the other hand, the map with its geometrically 
based spatial reference system is different from oral and written text, where 
we see a language-based spatial reference system. Written text loses its 
closeness to maps and gains a new closeness to oral text. This is not to 
deny virtual time in maps and virtual space in texts. However, these are 
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established at another level, based on the meaning of each expression. The 
space of the map and the time of the text is a primary level in the semiotic 
process of making meaning in the context of landscapes and embodied 
human experience. 

This implies that transmediation (Ellestróm 2020: 81-83) between oral 
and written texts, on the one hand, and maps, on the other, is slowed 
down by the differences between how the two media forms refer to exter- 
nal landscapes in their representational systems. Between oral and written 
texts, so different in their material interface and sense interaction, trans- 
mediation can be done with much less effort at the semiotic level. When it 
comes to integrating different media products in the same material inter- 
face, on the other hand, written text and maps integrate so well that it 
looks almost seamless, when maps illustrate textual documents as well as 
when texts add information to a perimap. Integrating oral text with either 
maps or written texts was much harder and drew more attention to the 
hybridity of the resulting media product, at least until the introduction of 
computer carried recorded sound. 

Transmediation between both is complex and will often be based on 
going back to the original story. This was clearly expressed for plastic arts 
and poetry already in 1766, when Lessing clarified the background for his 
suggestion that painting should be based on quite different principles 
from those underlying poetry: painting is about objects in space, and 
poetry about events in time (Lessing 1893 [1766]). The relationship 
between map/text studies and interart studies generally and Lessing spe- 
cifically is discussed further in Eide (2015), also indicating which of 
Lessing’s observations are still relevant and which are based on dated and 
now irrelevant aesthetic principles. 


7.7 . INTERPRETATIVE JOURNEYS 


There is an important additional aspect of spatial reasoning as it is expressed 
in texts and maps. The spaces we experience in landscapes, and are 
reminded of in media products, also form basic aspects of human thinking 
outside the actual navigation of landscapes. Two examples from classical 
rhetorics are the art of memory and the linearisation problem. We will also 
look into as if-travel in physics. 

The art of memory traces its roots back to Simonides (see Yates 1966: 
223) as a memory technique used in the preparation of speeches. Of spe- 
cial interest to us here is that its operational mode consists of linking a 
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landscape to a textual narrative, in this case oral. It is not about remember- 
ing every word exactly,? but rather about remembering the right order of 
the topics to be covered in the speech. One assumes, for instance, a garden 
and a walk from place to place. At each place, the speaker mentally stores 
objects reminding her of the upcoming topic. This has been shown to 
work as a memory technique across cultures and is still part of commer- 
cially available mind training packages. 

The starting point for the art of memory technique is an already created 
speech, and the spatial configuration is used to remember the main points 
in this speech. Thus, an oral or written text is transferred into a mental 
space, which is remembered and later used to re-create the original speech. 
Spatial memory is used as a mental storage technique. 

The linearisation problem is also known from classical rhetorics. The 
problem here is to create an oral or written text in the first place. How can 
one establish a linear text from a number of different topics which all need 
to be included? The solution here is to create a space where the topics are 
spread out, and then navigate this space like a walk. This walk is assumed 
to be linear in the same way as a speech (Levelt 1981) and it is important 
that the route passes through all of the topoi,'* that is, all the topics. In 
this case, it is not about remembering a text but creating it, out of a mass 
of relevant topics. 

In the development of thinking towards scientific hypotheses and even- 
tually theories that can be formalised in propositional language, active use 
of drawings is an important part of the building of mental models 
(Nersessian 2008). This has been documented as video recordings during 
social interaction among groups of physicians at different career levels 
(Ochs et al. 1994). 'The video footage shows a figure being drawn on the 
blackboard by one participant, then others adding their own marks to the 
drawing. These can either be unclear lines created by moving a finger over 
the blackboard drawing, or additional drawings with chalk. Furthermore, 
in the model space used to discuss particle phenomena, movement is 
described. Not just particles moving, but particles and other parts of 
nature are even addressed with words such as ‘me’ and ‘you.’ Thus, in the 
process of making sense of physical reality through the models used to talk 
about it, metaphorical travels through the model space—and presumably 
indirectly through micro-space—is done by humans in the room where 
the discussion takes place. This happens both through oral language— 
including gestures—and with the use of figures. 
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All these examples show the fundamental power and potential for 
meaning-making in thinking about space in a figurative way. This happens 
either as a space standing in for another space, however different it may be, 
or as a space standing in for a more abstract structure. Meaning is then 
expressed through stories about travelling through that space. This shows 
how the differences between figures, such as maps, on the one hand, and 
texts, oral and spoken, on the other, can be used creatively as a tool in 
developing, expressing, and negotiating meaning. As Simonson (2020), 
we see also here how media borders are pushed. Model-based thinking is 
fundamentally tied to the complexities in the relationship between texts 
and maps, as it passes the tunnel from senses and physical interfaces 
through space and time, real and represented, to a semiotics-based estab- 
lishment of meaning. 


7.8 | MODELLING AND MEDIA TRANSFORMATIONS 


The previous examples showed how visual forms are used in individual and 
social negotiations of scientific meaning in modelling processes. Indeed, 
what has emerged over the last years in digital humanities is a model con- 
cept which is in line with the concept of ‘media product’ in Ellestróm 
(2018: 11): “Since being a media product should be understood as a func- 
tion rather than an essential property, virtually any material existence can 
be used as one, including not only solid objects but all kinds of physical 
phenomena that can be perceived by the human senses.” The functional 
aspect of ‘model’ is made explicit through the focus on the verbalised form 
‘modelling’ rather than on the noun ‘model’. A model defines anything 
which is used by someone (a person, a group) in modelling. The physical 
item is important, but cannot determine if the process is a process of mod- 
elling, in line with the criteria for something being a technical medium of 
display (Ellestróm 2020: 33-40). 

A map is a media product insofar it is understood as a human-made 
expression intended to convey some sort of meaning to other humans. 
This leads to a question archaeologists struggle with, not only for maps 
but also for other forms of expression. How can one decide if lines on the 
wall of a cave are parts of a media product or not? The first attempt to 
clarify the question would be to identify the species behind the marks. If 
the marks are made by the claws of an animal they will not be seen as 
establishing a media product. But even if they are assumed to be human 
made, they might be the results of sharpening a tool, and not intended to 
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represent rivers and hills. If it is decided that the marks are made by a 
human and do indeed represent a landscape, then it is seen as a media 
product, and commonly described as a map. This map is a model of the 
landscape. 

This also applies to scholarly and scientific modelling—the same 
scratches could make up a model also in this sense. Imagine, for instance, 
two scientists in the late nineteenth century trying to understand the 
hydrological structures in the landscape around. Seeking refuge from the 
hot sun in the cave, they start discussing different hypotheses and eventu- 
ally, as they have left their writing utensils at the camp, use the wall and 
sharp stones to express different models, in line with the example from 
physics above. Scholarly and scientific modelling is an activity defined by 
who takes part in the activity and with which purposes. Thus, a media 
product is a model insofar as it is used in modelling, and it is a scientific or 
scholarly model insofar as it is used in scientific or scholarly modelling. 

The maps therefore represent models when they are used as models of 
a landscape. This happens in most intended use of maps, but not all. Maps 
are also decorative objects which can be used for other purposes than 
modelling. The modelling aspects are based on their representational 
function. 

But what is a map a model of? This question involves some additional 
complexities. From the outset, the map is a model of a landscape with 
some degree of realism. Some maps, however, are clearly presented as 
maps of a fictitious world, and are assumed to be so by most readers. A 
map of Tolkien's world is assumed to represent the same world as in the 
books written by Tolkien, and maybe also as in the films, the fan fiction, 
and other derived works. Such maps will be assumed to be more or less 
correct. Thus, there exists something that the truthfulness of a map of a 
fictitious landscape can be tested against. This is a shared conceptualisa- 
tion between the users of the text, testable against documents describing 
this world—in the case of Tolkien's cosmos, first and foremost his own 
work. A fictitious world can have an assumed internal truth system, fol- 
lowing some sort of spatial principles. It does not have to be coherent, 
but it is assumed to be truthful within the context of the fictitious work. 
The establishment of the intracommunicational domain (Ellestróm 2018) 
follows certain rules. 

A virtual world, as it is known through VR systems, adds another level 
to the experience of space. Independently from whether that virtual world 
is mediating a fictitious space or not, it does not only have to follow some 
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sort of spatial principles, but also the principles of physical sensation made 
by experiencing the known space. There is a specific type of embodiment 
present in this experience. As pointed out above: from the outset, the map 
is a model of a landscape with some degree of realism. VR might, at first 
glance, appear to present a model of space with a higher degree of realism, 
but what happens if a virtual space is not tested against the real physically 
experienced world, but rather against another media product? What if the 
virtual world is established through a media transformation (Ellestróm 
2020: 79-83) from, for instance, a text to a VR system? 

Going forward, we will focus on space as it is expressed and experienced 
in VR. On the one hand, space is crucial or even existential for media 
products which are experienced as VR. On the other hand, immersion is 
often seen as an inherent part of VR. Through analysing experimental 
media transformations, we will show that a transmission of the perception 
of space—which is as realistic as possible—only creates an illusion of a 
multi-dimensional space. This experience is therefore more connected to 
the traditional ways of expressing knowledge about landscapes (as it was 
described above) than one might expect. The linking between a media 
product that is conceptually creating space and the connection to be expe- 
rienced (also through traditionally mediated space as it was outlined 
above) is widening the discussion in new ways. 


7.9 LANDSCAPE, SPACE, REALITY, AND THE VIRTUAL 


We will now extend the discussion to three-dimensional modelling as a 
process of creating virtual worlds and accordingly VR systems. In such 
systems, spaces are expressed by computer screens, mobile phones, head 
mounted displays, or other technical media, enabling a virtual world. The 
way spatiality is expressed in these systems makes the representation of 
spatial relationships and landscapes, also morphing landscapes, accessible 
in ways not available through other technical media. Investigating VR as a 
medium for conveying experiences of landscapes opens up new perspec- 
tives on mediated space and time. Furthermore, examining how scholarly 
and scientific models are expressed in multi-dimensional space allows ana- 
lysing the spatial understanding, not only of the creator but also of the 
user. This has significant impact on how models can be used in scholarly 
and scientific work. 

The purpose of creating a 3D model is dependent on its intended use, 
but also connected to the medium to be used to visualise the model. 
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Displaying 3D models through VR devices and thus representing an inter- 
active and explorable three-dimensional computer generated environment 
(a virtual world) is usually associated with expectations deriving from a 
general conceptual understanding of VR as a specific type of reality emula- 
tion. It is more challenging to establish a conceptual definition of VR. It 
must be based on an understanding of the complexity of immersion, in 
line with Virginas’ discussion of the video quality of reproduced record- 
ings in Lynch’s Lost Highway (2020: 160-161). 

This complexity is linked to the special nature of embodiment in the 
use of VR equipment, and links to performance studies (Vieira 2020: 
8, 10). Maybe the experience of travelling can be extended to the context 
of architecture, beyond the possibility to see and move in non-existing 
buildings. This begs the question, however, of the vital role of embodi- 
ment in architecture and how far this can be re-created in VR. Can, in fact, 
other media forms experienced in VR, for instance, theatre, be seen as 
experiences of the represented media forms, given the differences in 
embodiment (Ellestróm 2020: 81-83)? 

Experiencing a virtual world is conceptually connected to the idea of 
being immersed within an artificial environment. It implies the possibility 
of strongly realistic user interaction with the virtual surroundings simulat- 
ing an experience, which gives the user an impression of being part of an 
illusionary perfect other reality. This other reality can mediate, for exam- 
ple, past events and distant places, but also entirely different, unprece- 
dented worlds. However, at this level, it is questionable how new VR is. 
The idea of realistically modelling an immersive representation is not only 
expressed in current development and implementations in VR but also 
expressed much earlier in paintings, optical illusions, and other forms of 
pre-digital media products. 

The dream of VR as it arises from the conceptual model presented 
above has never been fulfilled completely or even satisfactory for most 
users. There are limitations both technically and in terms of which graphi- 
cal representations are possible. In order to feign a fully realistic experi- 
ence of the unrealistic, a VR system would have to trigger all the sense and 
perception systems of the user. Existing VR implementations and devices 
nevertheless have a potential for representation which can be unlocked 
without insisting on a specific level of performance in the VR system. 

Setting aside the conceptual expectations and considering existing 
implementation of VR within the technical possibilities available today, 
one can explore in practice the actual possibilities for representation. VR 
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can be used for modelling highly realistic yet completely different reali- 
ties—completely different in the sense of different to any known place but 
still realised through representations of known objects and known physical 
characteristics. One example is SpaceVR, a project that attempts to make 
the universe explorable through VR. Such attempts apply analogously to 
many unattainable spaces in the world known to us. Accordingly, the 
attempt to make known, but physically inaccessible places explorable 
through VR is a frequently observable phenomenon.'” Photorealism is 
one important goal in VR, but other ways of representation are also 
important, including highly abstract forms; there are, for instance, several 
implementations in VR representing paintings by Hieronymus Bosch. 

Vieira (2020) established the role of the architect as a sort of a theatri- 
cal producer. Also in the analysis of VR, it is highly useful to include practi- 
cal experiences from the creation of VR systems. This balances the focus 
between conceptual ideas and possibilities on the one hand and the actual 
possibilities of implementation on the other. While it is important to plan 
VR implementations within the framework of existing technical possibili- 
ties, the creation of three-dimensional worlds with the help of computers 
additionally reflects the perception of known reality. As pointed out for 
other media above, the users will assume similarities with their known 
environments and only change their expectations when they have to. 

For the experience of VR, a virtual environment must be created. This 
environment consists entirely of three-dimensional objects. None of the 
visual characteristics from our known reality is self-evident. Everything, 
including reflecting light distinguishing between different materials (for 
instance, wood or plastic), shadows, and the appearance of highlights in 
white, is calculated by algorithms and can be influenced by the creator of 
the virtual world. Only what the user can see needs to be calculated, and 
when something can only be seen at a distance, it can appear realistic even 
if in lower quality. Consequently, there is no need to adapt the representa- 
tion to the user's world of experience, at least not when considering the 
possibilities for implementation. As in theatre and film sets, the world can 
be set up by invented backdrops. The artistic process of creation, in this 
case the modelling of virtual surroundings, always has artificial aspects 
which open up for new interpretations. 

The ability to experience space and to move in the virtual world is usu- 
ally, but not always, linked to different user interactions. Ultimately, 
almost every decision in the modelling process can be made by the mod- 
eller. Metaphors and experiences from known reality are often used; 


7 SEEING THE LANDSCAPE THROUGH TEXTUAL AND GRAPHICAL MEDIA... 197 


however, the way we orient ourselves can only partially be transferred. 
Orientation in an unknown space is always of key importance and will be 
of immediate concern to a new user. As we saw above, everybody has the 
experience of being in an environment, even if the experienced environ- 
ments vary significantly. The creation of VR systems is connected to a 
desire to share experiences, but also to create a world which cannot be 
experienced in reality based on the experiences the user already has. Such 
worlds can still be expressed through mediation, and, as pointed out by 
Simonson (2020), this also happens in other media than VR. The experi- 
ence of movement in a virtual environment establishes a direct reference 
to a reality understandable by humans. The ability for abstraction we saw 
through maps and text has its counterparts which can be analysed in the 
implementation of a VR. 

A commonly used case for VR is games, which are similar to video 
games known from other technical media. Even VR applications not 
meant to be games often use interfaces and elements known from video 
games. In the context of games, users interact with a so-called cyberspace, 
a completely virtual surrounding in which gamification aspects are not 
essential. The experiences of how space is represented in video games are 
used in a comparable way in VR. Maps are often important elements in 
this context, being used to discover the virtual world. The abstractions 
used in these maps are similar to the maps described above. The same 
language is used. Modelling a virtual world is often combined with design- 
ing maps, which is of essential importance to user interaction, communi- 
cating the space to the user. Besides this communication, characters can 
also talk to each other about places that are not covered by maps. 

The use of both maps and texts in video games is well established for 
communicating information about the spatial structure of the virtual 
world, and for telling stories to the user.'* Maps mostly function as naviga- 
tion tools: they show where you are and sometimes they also help users 
navigate, by showing them where they can go. Pathfinding and navigation 
are often parts of the users own decisions in the game. The map covers all 
places that are available, implying also that some places are not of interest 
or not reachable. One can design virtual worlds where the user can get 
lost. The double nature of maps as tools for overview and control and as 
tools for navigation (Wood 2010) is often present in the use of game maps. 

Places visualised on a map establish the expectation that they will in fact 
be reachable. The appearance of a map often fits the design of the virtual 
world it is describing. The places covered by the map are only references 
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and cannot in themselves establish a story around the places being referred 
to, or cover their meaning. Compared to descriptions of places in texts, 
the reference will be less likely to create emotions connected to the place 
(Eide 2015: 152-153). The experience of actually visiting the virtual place 
can overcome this. 

Video games and maps have a quite special relationship, as shown by an 
endless number of examples from the last decades. Similar to board games, 
maps can become a central element of the game also through functions 
that go beyond pure orientation.’ As a result, positions of players on the 
map are elementary in the sense that each location and spatial relationship 
on the map is important. At the same time, the respective positions must 
be visible and perceivable either by an opposing team or by other players. 
A live mapping of all objects of interest is possible in a virtual environ- 
ment, where also moving objects such as cars can be shown. The virtual 
world and its functions are established through maps. The map can func- 
tion as an interface, independent from the user. Sometimes the map itself 
is responsible for offering challenges. 

Groups of media computer science students at the University of 
Cologne have examined modelling processes of media transformations, 
partly through VR implementations and partly through attempts to model 
maps in VR (Eide et al. 2020). Analysing the results of the students! work 
allowed us to make some observations of the modalities of VR applica- 
tions. Given that specific media products in VR differ from each other, and 
the rapid technological development, the results are bound to be 
preliminary. 

We identified three general questions that pinpoint specific decisions 
made through the modelling. The questions and the students? answers are 
shown in Figs. 7.1, 7.2, and 7.3. Our observations for VR as a non-bodily 
media product can be summarised in the context of media modalities 
(Ellestróm 2020). The material modality as a latent corporeal interface of 
the medium, and how its different modes are expressed, depend partly on 
which type of VR device is used. The bodily movements of a user can in 
some cases be fully recognised through motion detection. The movement 
of the head is essential to control the point of view. The materiality of a 
controller, gloves, and a VR headset is normally established as demarcated. 
Less clearly demarcated material manifestations are expressed by artificial 
sounds and lights added to the virtual world. The perception of the inter- 
face of VR happens through the different senses, as modes in the sensorial 
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Say (voice) 


Show/ 
point to 


Fig. 7.1 How do you point your users to ‘interesting’ aspects of your 
VR-environment? Do you explicitly point your users to aspects relevant for your 
narration? 


modality. Feeling, tasting, and smelling are usually not included; however, 
seeing and hearing are attended through the VR headset with earphones. 

Space manifested in the material interface can be implemented through 
a map. Cognitive space is explorable through looking around in the virtual 
world. Virtual space is present in the virtual world. Time manifested in the 
material interface can be influenced by interaction to the extent it is imple- 
mented in the system. Perceptual time is established through interaction 
or animated camera movement. Virtual time is given through animations. 

VR uses a complex and many-faceted semiotic language. The basis for 
the photorealistic expressions one sees is iconic. In the image, however, 
there is always a set of symbolic elements. These can be graphical as well 
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Functional 


^ 


x 


Realistic 


Fig. 7.2 Do the objects in your VR-environment present your users a functional 
(symbolic) or a realistic view? 


as in the form of text, or as texts used as elements of interaction. Some 
parts of VR are close to map semiotics (MacEachren 2004; Brodersen 
2008), whereas other aspects trigger the indexical complexity of digital 
photography and video (Lister 2007). Being highly dependent on con- 
crete implementations, it is hard at this stage of our research to generalise 
the semiotics of VR. Further research is needed to establish how different 
the various technical implementations used for VR are semiotically, and 
whether it is possible to establish a general core understanding of VR 
semiotics. 
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Game 


^ 


Narration 


Fig. 7.3 Would you say your VR-environment is more a game or a narration? 


7.10 BEING IN A VIRTUAL WORLD, EXPERIENCING SPACE 
REAL AND UNREAL 


By putting on a theatre mask, the actress transforms into the role charac- 
ter.” In other theatre forms, the masks are invisible, but still metaphori- 
cally present. In all embodied, actor-based art forms, the actor meets the 
role character in a shared body, whose trajectory is both that of the role 
character and of the actor, as discussed also by Simonson (2020). In the 
1930s, the mask ofa gangster was gradually glued to the face of Humphrey 
Bogart, opening up for a new type of hero when the actor who had imper- 
sonated several criminals became the doubtful but inherently good guy in 
film noir of the 1940s. 
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When putting on a VR mask (a head mounted display), only the eyes 
are fully covered. This is necessary in order for the targets of light being 
emitted from two small screens, creating an illusion of a three-dimensional 
room. The earphones provide some replica of surrounding sound, while 
the sound of the physical space around the user of the VR system can be 
heard as distant, sometimes even unreal or dreamlike, dependent on the 
specific equipment used. This might be compared to the architecture of 
the theatre building (Crossley 2020: 96-97). Clarifying the relationship 
between the consequences of the different types of equipment and set- 
tings is beyond the scope of this chapter, but Crossley’s discussion of the- 
atre as a hypermedium could be extended by also looking at the theatrical 
text as basis for a VR application, in addition to VR within a theatri- 
cal event. 

As Alice experienced behind the looking glass, everything is the same 
yet everything might be twisted. In the end, the room on the other side 
turned out to be merely a dream. This is in line with literary functions, 
such as metalepsis, as pointed out by Lutas (2020), also for its counterpart 
in film. The idea of the computer game as life (Gaskill 2009) is a mirage, 
no more real than the dream of jazz music simplified to one single tone 
(Skagen 1993). Yet, it is a powerful argument ad absurdum, pointing 
towards the science fiction nightmare of the virtual that can kill you, in our 
century unified through the popularity of the Matrix trilogy. What can kill 
you is real and not just virtual. 

What we see in our VR worlds is both real and virtual. The lights and 
sounds are real. The 3D and surround effects are virtual. The avatars we 
meet can represent artificial intelligences on which our acts have no impact. 
They can also represent our best friend or partner, in which case mistakes 
done in the VR system can alter or even destroy lives. Like any other tech- 
nical medium, from parchment and paper to Instagram and Twitter, it can 
be used to send messages to the ones who do not care. It can be used for 
fully fictitious games, where all is virtual and nothing has consequences. It 
can also be used to communicate with others who rely on our words, 
images, and acts, in the realness of letters and images establishing or break- 
ing family bonds and starting or ending wars. The novelty of VR is not the 
fact that it is virtual. The virtual has always had a potential for linking to 
reality. The novelty of VR is its way to relate senses to a mix of real and 
fictitious environments. It hides the realness in a secluded space by being 
in a media artefact, and hides its virtualness though photorealistic 
immersion. 
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VR is not harmless. Books are not harmless. Whether we say that books 
kill, or we say that books are but tools used by humans in order to initiate 
acts that lead to killings, is a matter of where we stand. Nevertheless, what 
we say about books we can also say about VR. Behind all the differences 
between media types—behind their different ways of representing some- 
thing beyond themselves and behind the differences in time and space and 
mixes of them manifested in the material interface—they are all tools used 
to make the unreal real and the real unreal. The various media types are 
used to communicate across the abyss between human minds. As shown in 
Ellestróm (2020), the basis for all the discussions in this volume is 
communication. 

In spite of all the differences presented in some detail in this chapter, all 
these media forms can represent landscapes and events taking place there. 
None of them are the landscapes and events they represent. The differ- 
ences between their own landscape and soundscape are what establish 
them as different media, qualified and technical. The links to the landscape 
come about based on our knowledge of landscapes. The only thing the 
media can do is to remind us of the landscapes we know, reshuffling ele- 
ments in them. The rest is up to us. 


7.11 CONCLUSION 


This chapter focuses on how one can see the landscape through textual 
and graphical media products. This is important in the context of the pub- 
lication Beyond Media Borders: Intermedial Relations among Multimodal 
Media because the landscape itself and spatial thinking is a basis for human 
thinking and communication (Tversky 2019). The recipient’s embodied 
memories of experiences in an environment (real or virtual) are always 
connected to the decoding and understanding of spatiotemporal modes 
which are active in all media products. 

We have analysed different media expressions related to landscape, with 
a focus on spatial representations in different forms and on how transfor- 
mations can deepen our understanding. We started with written and oral 
texts and maps, and their combination in geocommunication systems, 
which are traditional ways of expressing knowledge about landscapes. By 
studying them as different systems for communicating spatial understand- 
ing, we clarified how each system works. The sequentiality of textual doc- 
uments is expressed in a spatial form, but is still recognisable based on the 
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code of writing systems, which is different from the other visual media 
we study. 

Interestingly, the spatial relationship between two places is established 
differently in texts—oral as well as written—compared to map documents. 
By focusing on place names as reference system, the different forms of 
spatial communication were highlighted: the geometrical reference system 
of the space of the map versus the primary syntactic relationships between 
words within a sentence. Opening up the discussion to spoken words also 
highlighted the importance of the human body as part of communication. 
Different media products are moreover concretely linked. The reference 
functions of maps and texts describing space do not just link media prod- 
ucts to landscapes, real or fictional; they also link media products, inter- 
nally or with other media products, through co-reference. 

Mental space, mental models, movement, and memories are basic 
building blocks in journeys, physical and mediated, with different levels of 
realism. Scientific modelling and fiction are different, yet connected 
(Suarez 2009). Thus, we were able to expand our research and discuss in 
a more abstract way how all this relates to modelling and modelling pro- 
cesses in general. By applying the concept of modelling from digital 
humanities and showing the connections with Ellestróm's concept of 
media product (2020), we could see that both are functions defined by the 
purposes of those who take part in the modelling and communicative 
activities. Thus, a media product is a model insofar as it is used in model- 
ling, and it is a scientific or scholarly model insofar as it is used in scientific 
or scholarly modelling. 

This analysis was then used to study concrete types of spatial expres- 
sions through new media types such as VR applications. Being new media, 
in a historical sense, means that they are less affected by conventions and 
have a much shorter tradition of developing their ways of using the avail- 
able affordances. Indeed, the affordances are still in formation and the 
technical development changes the affordances themselves. Moving 
beyond graphical two-dimensional expressions and written texts describ- 
ing landscapes into virtual three-dimensional ‘landscapes’ enables us to 
understand better the spatial media characteristics influencing media prod- 
ucts. An abstract versatility results from the existential basic property of 
VR spaces as they are used to create other spaces for users being in a virtual 
world, experiencing space real and unreal. This reflects on how space can 
be mediated and formed through experiences, memories, and illusions. 
Thus, landscapes in a number of different senses were discussed. We 
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started with the real environment that we move around in as embodies 
creatures and how it has been mediated over centuries and millennia of 
human culture, before shifting our attention to the landscapes we move 
around in through avatars, that is, VR. 

Our claim is that the relationship between the spaces in a virtual world 
and the spaces we know as embodied animals is complex and many-faceted, 
establishing VR both as mediated expressions (VR is a type of communica- 
tion device) and as experienced spaces. This is linked to an equally com- 
plex set of connections between different forms of embodiment and 
different sets of affordances. We need more research and more practice in 
this area to be able to clarify how all these parts interact in order to estab- 
lish human mediated experiences in communication. 


NOTES 


1. On the space/place distinction in philosophy, see Casey (1998). On space 
and embodiment, see, for instance, Gibson (1986) and Massey (2005). 

2. ‘Written text’ is here meant as verbal texts written in digital or paper based 
documents. *Map' refers to digital or paper based documents referring to 
an assumed external space through a geometrical reference system. For 
more details on the use of these terms, see Eide (2015). 

3. We omit here the complexities added by forms such as braille. 

. For a linguistic study, see Talmy (2005). 

. We will come back to the connections between maps and texts in more 

detail later. 

6. This is discussed further in Eide (2014). See also Eide (2015: 176-180). 

7. The speed of sound does introduce a slight lag in time. While this is impor- 
tant in some types of communication, for instance, between musical per- 
formers, it is rarely relevant to oral textual communication. 

8. The concept was established scientifically in Tolman (1948). 

9. Brodersen (2008) focuses on human communication. On animal commu- 
nication, see, for instance, Savage-Rumbaugh et al. (1998). 

10. It also links them to the gazetteer entry, which co-refers with each of the 

occurrences of the place name in the historical source text. 

ll. Examples include Tolkien’s works of fiction and Stephenson’s 

Treasure Island. 

12. See Harley and Woodward (1987) for a good example with a large number 

of map descriptions. 

13. This is different from the mode of memory assumed for Homeric epos, 

where sequences were remembered exactly. Rhythm and rhyme helped the 
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14. 


15. 


16. 


17. 


18. 


19. 


20. 


bard in fixating passages, which were then put together in series suitable 
for the performance (Havelock 1986). 

Greek topoi (sing. topos) means a place, region or a position. The English 
word ‘topic’ is a derivate through the adjective form. 

On models as media products and modelling as media transformations, see 
Eide (2015: 195-198). 

A good example of an incoherent landscape can be found in Ishiguro’s 
novel The Unconsoled (cf. Eide 2016). 

The possibility to discover real landscapes similarly to how we see them in 
real life has been connected to VR applications for a long time. A famous 
example is Google Earth VR. An early example is The Aspen Movie Map 
developed by MIT in 1978, which allowed the users a virtual tour through 
Aspen and could be seen as a VR version of Google Street View. 

A classic example is Doom. Doom VFR supports teleportation in addition 
to moving through the space of the game. The VR game contains new as 
well as classic Doom-Maps. 

Users are allowed to build their own virtual world within systems such as 
SimCity: Kingdoms and Castles, where also VR support is planned. 

This section is based on Eide et al. (2019). 
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8.1 SUMMARY 


In “The modalities of media II: An expanded model for understanding 
intermedial relations”, I proposed an extended and developed model—or, 
rather, a conglomerate of models—designed for systematic investigation 
and analysis of the basic features of all forms of media and their interrela- 
tions (Ellestróm 2020a). Instead of beginning with a limited set of estab- 
lished media types and their traits and interrelations, the model is founded 
on a delineation of the concept of media product and an explanation of 
media modalities—types of basic media traits—that are shared by all media 
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products and therefore also media types. This allows the model to account 
for the crucial fact that media products and media types are both similar 
and different. 

I also emphasised that the point of models is to put aside specific details 
to enable a view that is more generally valid. I want to offer a broadly 
applicable, well-developed and distinct but flexible theoretical framework. 
Although the model encourages methodical analysis of media and their 
interrelations, it does not offer a proper methodology. It suggests ways to 
tackle things but does not imply any fixed methods for investigation. 
Nevertheless, it is hopefully helpful for developing such methods and 
methodologies that must ultimately be formed based on one’s specific 
aims and goals and in the service of various research problems and ques- 
tions attaching to mediality at large and, more specifically, media 
interrelations. 

This was demonstrated in how the model was applied in the contribu- 
tions to the two volumes of Beyond Media Borders: Intermedial Relations 
among Multimodal Media. The chapters, written by scholars from various 
disciplines, tackled a broad range of basic and more or less established 
qualified media types, such as theatre, architecture, film (with or without 
embedded sound), live and online performances, audiobooks, written lit- 
erature, novels, book illustrations, posters, music, opera, comics, graphic 
novels, journalism, biographies, scientific articles and maps. The chapters 
also dealt with various sorts of moving pictures, a variety of radio and tele- 
vision programmes, and different forms of oral and written verbal language. 

Furthermore, the model was combined with a variety of complemen- 
tary theories and methods selected for dealing with the more specific issues 
of the chapters. Likewise, the chapter authors used it in the service of a 
range of different research questions, aims and goals. Mark Crossley 
(2020) investigated the challenging relationships between theatre and 
other qualified media types. Andy Lavender (2020) scrutinised how the 
work of actors and performers can be comprehended in terms of multimo- 
dality. Andrea Virginás (2020) analysed the representation of electronic 
screens in feature film, which has recently become common in film cul- 
ture. Chiao-I Tseng (2020) looked into the issue of story truthfulness and 
affective intensity in feature films in the light of how they use historical 
archive footage, found footage and various film techniques. From a con- 
text and user perspective, Iben Have and Birgitte Stougaard Pedersen 
(2020) researched how audiobooks are experienced and what it means to 
‘read’ an audiobook. Focussing on mobile language learning, Heather 


8 SUMMARY AND ELABORATIONS 215 


Lotherington (2020) explored multimodal encoding of contemporary 
digital mobile communication. Mary Simonson (2020) investigated his- 
torical and contemporary media products that communicate less straight- 
forwardly, partly because of their intermedial configurations. Kate Newell 
(2020) examined different forms of movement of pictures across media 
platforms and contexts, and the mechanisms that facilitate such move- 
ment. Based on thorough considerations of the modality modes of archi- 
tecture, Miriam Vieira (2020) analysed the presence of architecture in 
literature. Ana Munari Domingos and José Rodrigues Cardoso (2020) 
scrutinised the construction of ‘reality clues’ in journalism comics and 
biography comics. The aim of Jorgen Bruhn (2020) was to combine the 
ideas of ecocriticism with the analytical strength of intermedial concepts. 
Liviu Lutas (2020) scrutinised the broad transmediality of metalepsis, the 
paradoxical transgression of boundaries between logically distinct repre- 
sented worlds. Finally, Øyvind Eide and Zoe Schubert (2020) explained 
the various mechanisms that connect different media types representing 
landscapes. 

Whereas the authors used different parts and aspects of my model in 
their chapters, they also employed some central concepts that, in different 
ways, attach to the model in important ways without being properly devel- 
oped in “The modalities of media II” (Ellestróm 2020a). This is simply 
because doing so would lead too far and disrupt the structure of the 
already voluminous chapter; there are several other issues that would also 
deserve more extensive treatment. To slightly compensate for this, and 
hence further clarify the bonds between my theoretical framework, other 
broadly used concepts and the many chapters in Beyond Media Borders, 1 
will elaborate here on three of the concepts that were neglected in the 
opening chapter, although in other publications I have integrated them in 
the framework of my model. The first concept is adaptation, which several 
of my colleagues have discussed in their chapters. The second and third 
concepts are narration and language, which were present in most of the 
contributions to Beyond Media Borders in one way or another. Because of 
this omnipresence, I will concentrate on those two concepts. What follows 
here, therefore, is not at all a summary of the contributions to Beyond 
Media Borders; it is rather an enlargement of some of its conceptual nodes. 
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8.2 ADAPTATION 


In the context of humanities and mediality scholarship, adaptation is gen- 
erally understood as a phenomenon involving different media types (typi- 
cally theatre or written literature and film). Therefore, I have argued that 
adaptation should be seen as a form of transmediation (Ellestróm 2013, 
2017). I also briefly explored transmediation in *The modalities of media 
II" (Ellestróm 2020a: section on *Media transformation"). As stated 
there and elsewhere, I use the term ‘mediate’ to describe the process of a 
technical medium of display that realises presemiotic (potentially mean- 
ingful) sensory configurations. For instance, a piece of paper is able to 
mediate visual sensory configurations that are (once perceived and rudi- 
mentarily brought into semiosis) taken to be a food recipe, a bar chart, a 
scientific article or a musical score. If equivalent sensory configurations 
(sensory configurations that have the capacity to trigger corresponding 
representations) are mediated for a second (or third or fourth) time and 
by another type of technical medium of display, they are transmediated. In 
our minds, some of the perceived media characteristics of the target 
medium are, in important ways, the same as those of the source medium; 
they form a recognisable virtual sphere. This enables us to think that the 
musical score that is seen on the paper is later heard when it is transmedi- 
ated by the sounds of instruments. In other words, the score's vital char- 
acteristics are represented again by a new type of sensory configuration 
(not visual but auditory signs) mediated by another type of technical 
medium (not a piece of paper emitting photons but sound waves gener- 
ated by musical instruments) (Ellestróm 2014: 20-27). 

Just as there are many different media types and communicative situa- 
tions, there is a plenitude of transmediation varieties. Although this has 
the potential to make the study of transmediations an exceedingly broad 
field, research has only paid full attention and given names to a few forms 
of transmediation. In the study of art forms, the general term for transme- 
diation of media products to other media products is ‘adaptation’. 
However, for good reason, researchers such as Kamilla Elliott have argued 
that, in effect, adaptation studies must consider not only transfer of repre- 
sented media characteristics between single media products, but also 
among media types in general (Elliott 2003: 113-132). This is because 
one cannot normally reduce vital media interrelations to the connection 
between only one source media product and one target media product. 
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Nevertheless, one does not even tend to call all types of transmediation 
of specific media products ‘adaptations’. For instance, one seldom refers to 
transmediations from written, visual and symbolic (verbal) media products 
to oral, auditory and symbolic (verbal) media products—that is to say, the 
reading aloud of texts, or the other way around—as adaptation (however, 
see Groensteen 1989). The same goes for transmediations from non- 
temporal to temporal visual and iconic media products—from still to mov- 
ing images (as discussed by Dalle Vacche 1996)—and for transmediations 
from written, visual and symbolic (verbal) media products to oral, audi- 
tory, iconic and symbolic media products; that is, the ‘setting’ of text to 
music (Lin and Chiang 2016). We also generally exclude transmediations 
of media types such as libretti, scores and scripts from the domain of 
adaptation. 

Instead, the archetypical adaptation is a novel-to-film transmediation. 
However, few scholars have delimited the concept of adaptation exclu- 
sively for this specific type of transfer. Adaptation studies today frequently 
work with not only theatre, literature and film but also qualified media 
types such as computer games, opera, comics and graphic novels. This is 
the case in, for instance, Linda Hutcheon's A Theory of Adaptation 
(2006). The authors in Beyond Media Borders who write about adaptation 
attach to the general idea of transmediation and focus on adaptation from 
novel to feature film (Lutas 2020), from verbal biography to comics 
(Domingos and Cardoso 2020), from opera to film (Simonson 2020) and 
from novel to illustrated and graphic novels, television and radio pro- 
grammes, ballet, opera, theatre and film (Newell 2020). However, 
Hutcheon and the authors in this volume only address media with devel- 
oped narration, which is a common way of implicitly delimiting the notion 
of adaptation. 


8.3 NARRATION 


As stated (Ellestróm 2020a: section on *Heteromediality and transmedial- 
ity"), transmediality means that media products and media types may (or 
actually do) communicate equivalent things; they can, to some extent, 
represent the same or similar objects (in Peirce's sense of the notion). This 
implies that there may be transfers in time among media. Because narra- 
tion is transmedial in the strong intermedial sense that it transgresses basic 
as well as qualified media borders, many narratives can be transmediated: 
their characteristics may be represented again by different media types 
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and, despite the transfer, be perceived to be virtually the same. For many 
adaptation scholars, the transmediation of narratives is indeed the central 
concern. However, one does not have to refer to one's work as adaptation 
studies in order to examine transmediation of narratives. Transmediation 
of narratives is exceedingly common, not only in everyday communication 
but also in more complex and official systems of communication such as 
education, research and legal processes. It also flourishes in religion, art 
and entertainment. 

I have developed my concept of transmedial narration (Ellestróm 2019) 
from Marie-Laure Ryan's work on transmedial narratology (Ryan 2005), 
but framed it within a broader concept of communication and attached it 
to the model of media modalities. Circumscribing this, one must first note 
that narration does not exist by itself; it happens when we communicate 
with each other. Consequently, narratives and stories are not something 
that we find floating around independently, but something that minds 
communicate. Therefore, one can and should compare narration with 
other forms of communication. Narration is a communicative form that is 
specific and important enough to deserve special attention. At the same 
time, it is only a variation of, and in practice sometimes not at all clearly 
delimited from, producing virtual spheres in general. 

Nevertheless, I have proposed to define a narrative as a virtual sphere, 
emerging in communication, containing events that are temporally related 
to each other in a meaningful way. Thus, the core of a narrative is exactly 
this: represented events that are temporally interrelated in a meaningful 
way. As this core consists of several elements, one could also describe it as 
a scaffold. I have also suggested that a whole virtual sphere containing 
such a core and normally also other media characteristics should be called 
a narrative and that the scaffolding core should be called a story. Thus, one 
can simply describe narration as the communication of narratives. 

It follows from this that what one perceives to be the same story may 
be realised in dissimilar settings in different narratives. What one recog- 
nises as the same story can be narrated in different ways. For anyone 
acquainted with areas such as literature and film narratology, this conclu- 
sion does not come as a surprise. However, scholars have debated the 
nature of the sameness of stories. For transmedial research, the crucial 
point is that it is possible, common and often useful to perceive that vital 
core constituents of some narratives—certain events being temporally 
related in certain ways—are more or less similar to vital core constituents 
of other narratives, possibly represented by other media types. This is 
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scrutinised particularly by the authors in Volume 2, Part I of Beyond Media 
Borders, “Media Transformation” (Simonson 2020; Newell 2020; Vieira 
2020; Domingos and Cardoso 2020; Bruhn 2020; Lutas 2020; Eide and 
Schubert 2020). 

Given these conditions for transmedial narration, one must also empha- 
sise that stories may either be construed for the first time by the perceivers 
of media products (because of salient structures emerging as the narratives 
develop in the mind) or recognised (from earlier encounters with narra- 
tives or events in the world). In other words, the story may be based either 
mainly on intracommunicational objects arising in the virtual sphere, or 
on extracommunicational objects in the form of already known stories or 
perceived events. In any case, stories have no autonomous existence 
despite what one might be led to believe by certain narratological discus- 
sions. Stories are always results of some sort of interpretation performed 
by certain people in particular communicative circumstances—never 
objective existences, but possibly intersubjectively construed (cf. 
Thon 2016). 

The theoretical distinction between a complete narrative and its scaf- 
folding core story is essential for understanding transmedial narration: sto- 
ries are embedded in narratives and, to a certain extent, may be realised by 
dissimilar media. However, the surrounding narratives and the represent- 
ing media products are often conflated in narrative theory and sometimes 
termed discourse (but not by Seymour Chatman in 1978: 23-24). Yet, 
there are not only two levels here—called, for instance, story and dis- 
course—but rather three (cf. Genette 1980 [1972]; Bal 2009 who also 
suggested three-layer distinctions, although quite different from mine). I 
suggest that (1) a media product with particular basic media traits and 
other formative qualities provides certain sensory configurations that are 
perceived by someone; these sensory configurations come to represent (2) 
media characteristics forming a complete narrative with all its many spe- 
cific details and features; furthermore, the perceiver comprehends that this 
narrative surrounds (3) a scaffolding core, the story, consisting of repre- 
sented events that are temporally interrelated in a meaningful way. 

Based on these definitions and distinction, one can more clearly describe 
why stories and parts of their surroundings in the whole narrative may 
often be realised fairly completely by several kinds of media. It is because 
many media types have the capacity, to some extent, to represent events, 
temporal relationships, meaningful relationships and an abundance of 
other media characteristics. The story is normally only one of several 
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transmedial media characteristics in narratives. The complete narrative of 
a certain media product can include a multitude of different media charac- 
teristics that may be more or less transmedial. However, as a rule, a story, 
consisting of the essential temporal structure of a narrative, is more trans- 
medial than the complete narrative, although probably never wholly 
transmedial. 

Considering that narration involves comprehension of represented 
events that are temporally interrelated in a meaningful way, it is clearly a 
cognitive process that depends on collateral experience and, more specifi- 
cally, often on cognitive schemata. Sensing interrelations to be meaningful 
is at least partially a question of being able to relate them to things with 
which one is already familiar. Narratives are not objectively construed but 
depend on the mind’s inclination to form gestalts. 

However, it is ultimately the more inherent factors of media products 
that trigger the mind-work of communication and, to some extent, deter- 
mine how and to what degree various media forms may realise narration. 
It is clear that the same perceiving mind, harbouring a certain set of 
knowledge, experiences, values, memories and schemata, will interpret 
different media products in very different ways even if it perceives them in 
comparable circumstances. This is obviously because the media products 
are unalike in various ways and because the divergences are highly rele- 
vant. In order to understand how dissimilar media types can communicate 
narratives, one must scrutinise the fundamental similarities and differences 
among media types and the extent to which these differences in modality 
modes matter (Ellestróm 2019: 53-58, 2020a). 

Although differences in modality modes are largely responsible for dif- 
ferences in the kind and degree of narration in various media forms, exam- 
ining them does not offer a convenient shortcut to full understanding. 
Thinking in terms of media modalities is not a quick fix. The basic prese- 
miotic and semiotic traits are always embedded in complex surroundings, 
which means that they generally need to be analysed in their interactions 
with each other and with additional factors. Nevertheless, modelling nar- 
ration in terms of media modalities facilitates a methodical approach to the 
issue of transmediality. Having different material, spatiotemporal and sen- 
sorial modes implies having partly dissimilar capacities for narration and, 
similarly, the use of different sign types has consequences for narration. 

The material modality is perhaps the least crucial category of media 
traits for determining narrative capacities. Solid media products such as 
written verbal texts, as well as non-solid media products such as spoken 
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verbal texts, clearly have very high narrative capacity, as decades of intense 
research has demonstrated. Furthermore, organic media products such as 
moving human bodies, as well as inorganic media products such as dolls in 
motion, may form complex narratives. 

The spatiotemporal modality is much more critical for narration because 
the scaffolding core of narratives consists of represented events that are 
temporally interrelated. The key issue then becomes the extent to which 
the representation of a temporal object requires a representamen with cer- 
tain spatiotemporal qualities. There is not much to indicate that media 
products should have specific spatial traits in order to be able to narrate 
successfully. Moving human bodies and dolls in motion are three- 
dimensional and suitable for narration. Written verbal texts are two- 
dimensional but also have the potential to be superbly narrative media 
products. Spoken verbal texts emanating from a singular source are spatial 
only in a limited way, but are still well suited for narration. 

However, there are some relevant differences between temporal and 
static media products. Moving images that are inherently temporal may 
effortlessly represent sequences of events and form elaborate narratives (cf. 
the discussions of narration in silent film, sound film, opera, theatre and 
performance in Lavender 2020; Lutas 2020; Simonson 2020; Tseng 
2020; Virginás 2020). This is not to say that one must understand the 
represented events to be interrelated in precise accordance with the tem- 
poral unfolding of the media product. In contrast, still images are, by defi- 
nition, static and therefore incapable of representing events that one 
inescapably perceives in a certain temporal order. However, this is not the 
same as being incapable of representing temporally interrelated events. It 
only means that the scope of possibly represented events is reduced 
(assuming that the size of the still image is not huge) and that the percep- 
tion of possibly interrelated represented events is not strongly directed by 
the physical interface of the media product (cf. the analysis of narration in 
architecture, a non-temporal and very much iconic qualified media type, 
in Vieira 2020). 

Nevertheless, the difference in spatiotemporal modes reduces the nar- 
rative potentiality of still images compared to moving images—at least if 
one considers media products constituted by single still images. However, 
it is possible to construe media products consisting of a whole set of still 
images. In itself, this does not enhance the narrative capacity, but it does 
open the way for the use of a special kind of symbolic element, namely the 
convention of sequential decoding. Perceivers who have learnt to process 
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parts of certain kinds of static media products in a regulated order may 
distinguish represented events in temporal sequences that are as stable as 
those produced by physically temporal media products (as demonstrated 
in the narratological analyses of illustrations, comics and graphic novels by 
Domingos and Cardoso 2020 and Newell 2020). 

This line of reasoning is also applicable to the difference between spo- 
ken verbal texts and written verbal texts: the distinction between temporal 
and static media products cuts through both images and verbal texts. 
Spoken verbal texts are temporal because the sensory configurations of 
such media products change constantly. Written verbal texts are static 
because the sensory configurations of such media products remain the 
same from one moment to the other (unless, of course, one perceives the 
text while it is being written or is a part of a temporal, visual media prod- 
uct such as a film). This means that spoken verbal texts, just like moving 
images—given that one allows for a certain volume of temporal exten- 
sion—readily represent sequences of events and may therefore produce 
intricate narratives (as seen in the analysis of audiobooks and radio speech, 
respectively, by Have and Pedersen 2020 and Simonson 2020). In con- 
trast, written verbal texts are normally static and if we think of written 
verbal texts in rough analogy with solitary still images—namely as consist- 
ing of single entities such as one letter or one word—written verbal texts 
are equally handicapped when it comes to representing events that are 
inevitably perceived in a certain temporal order. 

In the case of language, however, the convention of sequential decod- 
ing is so strong that one normally understands written verbal texts as con- 
sisting of large sets of subordinate symbols that are bound to be decoded 
in a highly regular manner. As in the case of sequential decoding of still 
images, this can lead to the discernment of represented events that are 
temporally interrelated in a manner that is as stable as those formed by 
physically temporal media products (cf. the discussions of narration in 
non-temporal, language-based media types such as various forms of writ- 
ten literature [Have and Pedersen 2020; Lutas 2020; Newell 2020; Vieira 
2020] and popular scientific articles [Bruhn 2020 ]). This is why so many 
researchers—misleadingly, I would argue—have claimed that written ver- 
bal texts are temporal. Such a conception obscures the difference among 
the physical appearance of representamens (the traits of media products), 
the process of perceiving the physical appearance of representamens, and 
the virtual appearance of represented objects (the traits of virtual spheres). 
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Thus, the fact that one perceives all kinds of media in time has some 
bearing on their capacity to represent temporally interrelated events: con- 
ventionalised orders of decoding may strongly enhance the narrative 
capacity of static media types. However, this does not erase the substantial 
differences between inherently temporal and static media. 

The sensorial modality also plays a role for the narrative capacity of 
media products. This is mostly because the senses (understood here as the 
external senses) are not developed cognitively to the same degree. Sight 
and hearing are our two most advanced senses, in that they are strongly 
connected to complex cognitive functions such as knowledge, attention, 
memory and reasoning. This means that sight and hearing are both well 
suited for narration. Indeed, all contributions to Beyond Media Borders 
that engage in narration deal almost exclusively with visual and auditory 
media types. 

However, this does not exclude in principle the other senses. One may 
use the faculty of touch for reading braille, for instance, or sensing the 
forms of reliefs and three-dimensional figures forming narratives. It is also 
fully possible to consider successions of interpersonal touches that form 
casual, narrative media products. Children playing and adults having sex 
may well communicate elementary narratives by way of sequences of 
touches that are performed and located differently. 

I presume that it would also be possible, in principle, to construe lan- 
guage systems mediated by taste or smell. In practice, however, they would 
probably be inefficient, as a speedy decoding of symbols requires quickly 
performed sensory discriminations. However, one can use taste and smell 
to create at least rudimentary narratives. A well-planned meal with several 
courses served in a certain order may be construed as narrative, to the 
extent that tastes and taste combinations may be developed, changed and 
contrasted in such a manner that gives a sense of meaningfully interrelated 
events. A series of scents may be presented in such a way that represents, 
say, a journey from the city through the woods and to the sea, including 
encounters with people and animals with smells that reveal certain 
activities. 

The three main modes of the semiotic modality are iconicity (based on 
similarity), indexicality (based on contiguity) and symbolicity (based on 
habits). All of these semiotic modes are immensely important for the reali- 
sation of narration. The majority of the more acknowledged basic media 
types that are commonly reasonably well defined and have accepted names 
in ordinary language are dominated by iconicity or symbolicity. One can 
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clearly characterise most of the recent examples of potentially narrative 
media types by a semiotic hallmark. Verbal texts, whether they are visual, 
auditory, or tactile, rely heavily—although certainly not exclusively—on 
symbolicity: the conventional meaning of letters, sounds, words and so 
forth. Moving and still images, whether they are visual, auditory or tactile, 
are understood to signify primarily through iconicity, based on perceived 
similarities between representamens and objects. Although series of 
touches, tastes and scents are hardly acknowledged as media types in com- 
mon parlance, one could make a case for recognising them as basic media 
types dominated by indexicality: real connections between the perceived 
sensory configurations and what they stand for. 

For the sake of clarity, I have tried to isolate the possible contributions 
of various media modes to narration. By highlighting modal differences, it 
is possible to discern media traits that contribute to narration existing in 
different degrees. However, media products are normally more or less 
multimodal—in very different ways—which makes the above generalisa- 
tions fuzzier, the differences among media types more subtle and the issue 
of transmedial narration more multifaceted. The model of media modali- 
ties does not offer a lexicon of transmedial narrative capacities as much as 
a methodical approach to examining narration in a wealth of dissimilar 
media products and media types. In each specific media product and media 
type, the present modes of the modalities add, in profound interaction, to 
the forming of virtual spheres and possibly narratives. In a certain media 
product, all of the various presemiotic modes contribute to forming cer- 
tain sensory configurations: a cluster of physical representamens that 
together come to represent —iconically, indexically or symbolically—a cer- 
tain cluster of objects that possibly forms a narrative. 

Therefore, I support Karin Kukkonen's conclusion that “[i]f, with 
Ryan, we understand narrative as a cognitive construct, different modes in 
multimodal media work together to provide the reader with clues to fill 
gaps and formulate hypotheses? (Kukkonen 2011: 40). Importantly, how- 
ever, I go beyond the rather coarse notion of mode used by Kukkonen and 
in so-called social semiotics in general: modes understood as text, image, 
gesture and so forth. For me, multimodality is a more fine-grained con- 
cept that can be more precisely circumscribed as four kinds of multimodal- 
ity: multimateriality, multispatiotemporality, multisensoriality and 
multisemioticity. As a rule, actual media products and media types have 
many modes of the same modality. For instance, media products that 
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consist of both organic and non-organic materiality are multimaterial. 
Media products that are both spatial and temporal are multispatiotempo- 
ral. Audiovisual media products are multisensorial. Furthermore, many 
media are multimodal in several ways simultaneously. 

Finally, most media products are multisemiotic to the extent that sign 
types typically work in collaboration. In an early article advocating the 
value of applying Peircean semiotics to the study of narratives, Robert 
Scholes suggested, “we cannot understand verbal narrative unless we are 
aware of the iconic and indexical dimensions of language” (1981: 205), 
which is certainly true. Even though symbolic signs are clearly the most 
salient ones, verbal language does not work solely through symbolicity. In 
visual language, for instance, elements such as lineation, letter size, letter 
form and empty spaces may create iconic meaning. In auditory language, 
iconicity is often produced by certain sound qualities, intonations, rhythms 
and pauses. By the same token, most media types signify through iconicity, 
indexicality and symbolicity in combination, although they are typically 
dominated by certain kinds of sign functions. However, one can find 
instances of communication and narration characterised by such extreme 
multimodality that virtually all kinds of modality modes, both presemiotic 
and semiotic, are included. 


8.4 LANGUAGE 


Having emphasised the multisemioticity of language and noticed that lan- 
guage may be visual as well as auditory, it is time to get into the details of 
the concept of language itself ( Ellestróm 2020b, forthcoming). How does 
it fit into the theoretical framework presented in *The modalities of media 
II: An expanded model for understanding intermedial relations" ( Ellestróm 
2020a)? Should the concept be understood as a media type? Not really. In 
her contribution to Beyond Media Borders, where she investigates a broad 
range of language forms in the context of language learning, Heather 
Lotherington accurately notes that *[l]anguage has traditionally been 
described as a medium: of communication, and of learning”, although 
*[1]anguage is, in fact, an abstract until it is materialized: mediated physi- 
cally, in speech and signed conversations, and technologically, in printed 
documents, social media sites, roadside signs, movies, games, and such- 
like" (Lotherington 2020: 219). 
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If language is an abstraction, then how should it be circumscribed? In 
line with a common practice, I suggest that language might be compre- 
hended as a communicative sign system. Given that a system is something 
that is highly organised, a communicative sign system must rely strongly 
on robust habits or, in other words, conventions. In terms of Charles 
Sanders Peirce’s semiotic vocabulary (Peirce 1932: CP 2.297 [c. 1895]), 
this reliance suggests that communicative sign systems are symbolic sign 
systems. Therefore, I define a language as a system of symbolic signs used for 
communication. 

Given such a definition, it is clear that there are several forms of lan- 
guage. Apart from so-called verbal language, humans have developed a 
plenitude of symbolic sign systems involving a broad variety of signs, 
such as mathematical and logic symbols, maritime signal flags, declama- 
tory gestures in theatre and even flowers that communicate socially 
through their kinds, colours and arrangements. Symbolic sign systems 
may consist of anything from thousands of symbols and very elaborate 
and detailed conventions to considerably fewer symbols connected more 
roughly through limited sets of habits. In their contribution to Beyond 
Media Borders, Domingos and Cardoso (2020) thoroughly analyse the 
combination of verbal and nonverbal languages in comics, involving both 
highly structured visual verbal language (involving conventional use of 
letters, punctuation, grammar, etc.) and less formalised nonverbal lan- 
guage (based on the habits of forming and combining visual, static 
images). 

Verbal language, understood as language involving features such as 
words and grammar, is based on extensive and very systematic conven- 
tions. However, verbal language is not really ‘one’ system, even disregard- 
ing the crucial differences between language forms such as Hindi, Spanish 
and Afrikaans. Some decades ago, Jan Mulder argued in detail for the 
notion that written and spoken languages constitute separate semiotic sys- 
tems. Based on the simple but far-reaching observation that what is seen 
(written words) is different from what is heard (spoken words), Mulder 
argued that it is “terribly wrong” to see written and spoken languages as 
“mere variants of the same thing”; they are “entirely different semiotic 
systems” (Mulder 1994: 43). I think this idea is correct, even though the 
two different systems of symbols have been developed to allow for 
advanced mutual transmediation: changing from writing and reading to 
speaking and listening is often an efficient although definitely not seamless 
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procedure. Hence, in more general discussions, it is fully comprehensible 
to treat verbal language simply as a common system of symbols (as Vieira 
2020). At other times, however, it is crucial to be more specific about how 
communication is realised, which requires specifying whether written lan- 
guage (Bruhn 2020; Lutas 2020; Newell 2020), spoken language 
(Lavender 2020; Simonson 2020) or both (Eide and Schubert 2020; 
Have and Pedersen 2020) are at stake. 

There are actually at least three different verbal systems of symbols. 
Sign language also interconnects strongly with written and spoken verbal 
language, although it is clearly a sign system of its own and not only a vari- 
ant of the same thing. Each of these three or more systems can be subdi- 
vided into even more specific systems such as written Swedish, spoken 
English or Chinese sign language. 

Although the definition of language as systems of symbolic signs used 
for communication captures the core of languages, it is naturally not an 
exhaustive description of how language can be conceptualised. Whereas 
written verbal language, my primary example here, is certainly always per 
definition based on systems of symbols, written verbal language may also 
include other sign types. In accordance with Peirce’s notion of sign types 
always being mixed (Peirce 1932: CP 2.302 [c. 1895]), there are probably 
also always at least some elements of iconicity and indexicality in semiosis 
dominated by symbolicity. Depending on factors such as history and cul- 
ture, nonprofessionals and scholars may see these other sign types as more 
or less inherent or alien facets of what they understand to be normal writ- 
ten verbal language. Even though one can give written verbal language a 
transhistorical and transcultural core definition in terms of symbolic sign 
systems, it is clear that its realisations vary considerably in relation to time 
and space. 

To exemplify this, I will focus here on the presence of iconicity in writ- 
ten verbal language. Iconic meaning-making is not based on habits or 
conventions, as is the case for symbolic meaning-making. Iconicity is 
grounded in perceived similarities among sensory perceptions and cogni- 
tive structures. In the domain of written verbal language, seen from a 
Western perspective, this means that symbols such as letters, words and 
punctuation, as well as spatial arrangements of these, may to some extent 
produce meaning. This is not only because of learned, habitual connec- 
tions between the visual impressions and what one takes them to repre- 
sent, but also because of perceived similarities between what one sees and 
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what the visual impressions are understood to signify. For instance, visual 
empty spaces between words and sentences suggest semantic spaces or dif- 
ferences. A more blatant example can be provided with a sentence like this: 
‘I can see two MOOns.’ 

Most Western scholars have long been strongly inclined to dismiss ico- 
nicity as, at best, only a peripheral part of verbal language, oral as well as 
written. Since the spread of Ferdinand de Saussure's linguistic dogma of 
the arbitrariness of language in Course in General Linguistics a century ago 
(2011 [1916 ]) belief in iconicity's existence in or relevance for language 
has sometimes been considered naive and has even been subject to scorn 
and ridicule. 

However, this has not been the case for Eastern scholars. According to 
a recent article by Ersu Ding, Saussure's quick dismissal of iconic signs in 
language as peripheral and linguistically uninteresting has been met with 
much greater scepticism in China (Ding 2014: 121). As Ding succinctly 
put it, “his holistic scheme of sign formation simply could not accommo- 
date anything other than a systemic pairing of the signifier and the signi- 
fied as a result of structural differentiation" (Ding 125). In China and 
other Eastern countries, such as Japan, the basic concept of language, 
including written verbal language, has thus been more inclusive—even 
though the core of language as a system of symbolic signs used for com- 
munication has not been denied, to the best of my knowledge. Therefore, 
one could dare to say that the concept of written verbal language in the 
West and the East has largely been the same, but not really the same. In 
any case, there is now massive support from empirical research, from both 
the West and the East, that iconicity, in varying degrees, is a solid compo- 
nent of verbal languages from all over the world. 

What, then, is the relation between the two concepts of language and 
medium? As I define it, a media product is a material entity—or, more 
broadly, a physical, intermediate entity—that makes communication 
among human minds possible (Ellestróm 2018). Thus, media products 
are material objects, actions or processes that are indispensable for con- 
necting human beings mentally. Consequently, they must be conceptual- 
ised in terms of both physicality and cognitive capacities. In other words, 
media products have both pre-semiotic traits and semiotic traits, meaning 
that they manage to trigger meaning-production in various ways based on 
their material, spatiotemporal and sensorial traits. 

Furthermore, nonprofessionals and scholars tend to group media prod- 
ucts into categories. People sometimes pay attention mainly to the most 
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basic features of media products and classify them according to their most 
salient material, spatiotemporal, sensorial and semiotic properties. For 
instance, we think in terms of still images (most often understood as tan- 
gible, flat, static, visual and iconic media products). Still images are an 
example of what I call a basic medium (a basic type of media product). 

However, such a basic classification is sometimes not enough to capture 
more specific media properties. We then qualify the definition of the media 
type that we are after and add criteria that lie beyond the basic material, 
spatiotemporal, sensorial and semiotic properties; we include all kinds of 
aspects regarding how the media products are produced, situated, used 
and evaluated in the world. One may wish to delimit the focus to still 
images that, for example, are produced with the aid of cameras—photo- 
graphs. Photographs are an example of what I call a qualified medium (a 
qualified type of media product). 

In contrast to media types, which involve pre-semiotic and semiotic 
traits, I comprehend languages somewhat more narrowly as systems of 
symbolic signs. From these dissimilar definitions, it follows that one can- 
not simply equate languages with media types. Media are material, and, as 
Lotherington puts it, language is something abstract “until it is material- 
ized: mediated physically” (Lotherington 2020: 219). However, lan- 
guages and media types are clearly interconnected. One way of putting it 
is to say that a language can be a vital semiotic part of a media type. For 
instance, language is an essential part of email and telephone conversations 
as well as of theatre and stand-up comedy. Put another way, specifying the 
definition of a language from an abstract symbolic sign system to the con- 
crete use of a symbolic sign system brings you closer to understanding the 
language as a media type. Fully concretising how a symbolic sign system is 
mediated—that is, how it is realised in specific material, spatiotemporal 
and sensorial ways—makes a symbolic sign system equivalent to a media 
type, as I conceive it. 

This equivalence is actually partly achieved in the distinction between 
at least three different verbal sign systems mentioned earlier: written, spo- 
ken and ‘signed’ verbal language. The notion of written verbal language 
definitely includes a specific sensorial mode, visuality, and often a spatio- 
temporal mode: the visual signs being realised on a flat surface. Likewise, 
the notion of spoken verbal language definitely includes a specific senso- 
rial mode, audibility, which requires materiality capable of transmitting 
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sound waves and temporal extension. The notion of ‘signed’ verbal lan- 
guage in effect fully covers the criteria for being understood as a media 
type: sign language is mediated by human bodies acting in time and 
three-dimensional space to produce movements that form visual signs; 
these signs are clearly part of a verbal symbolic sign system involving 
strong facets of iconicity and indexicality. Considering that sign language 
is also designed specifically for face-to-face communication with and 
among people with hearing impairments, it is a qualified rather than a 
basic media type. Other fully defined verbal sign systems include Braille 
writing, designed for tactile verbal communication with people with 
visual deficiencies and, by the same token, a qualified rather than a basic 
media type. 

However, one can hardly describe ‘normal’ written verbal language 
(based on vision) as a qualified media type. Instead, I would say that this 
category includes several closely related basic media types. As mentioned, 
written verbal language is definitely visual and normally realised on flat 
surfaces, but it can also occur on rounded and irregular surfaces, and let- 
ters and words may be three-dimensional in themselves. Written verbal 
language is most conveniently mediated through solid and inorganic 
materiality, but it is fully possible to write in gas and liquids or with the use 
of organic material such as living bodies. Thus, written verbal language is 
quite an open category that includes a variety of basic media types, depend- 
ing on whether the more exclusive Western or more inclusive Eastern 
comprehensions are included. 
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